A  General  Framework  for  Flow  Control  in  Wireless 

Networks 


Minghua  Chen 


Electrical  Engineering  and  Computer  Sciences 
University  of  California  at  Berkeley 


Technical  Report  No.  UCB/EECS-2006-194 

http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-194.html 

December  22,  2006 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

22  DEC  2006  2  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2006  to  00-00-2006 

4.  TITLE  AND  SUBTITLE 

A  General  Framework  for  Flow  Control  in  Wireless  Networks 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROTECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

University  of  California  at  Berkeley, Electrical  Engineering  and 

Computer  Sciences, 387  Soda  Hall, Berkeley, CA, 94720-1776 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

see  report 

15.  SUBIECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

_ _ _  ABSTRACT 

18.  NUMBER  19a.  NAME  OF 

OF  PAGES  RESPONSIBLE  PERSON 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  Same  OS 

unclassified  unclassified  unclassified  Report  (SAR) 

161 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


Copyright  ©  2006,  by  the  author(s). 

All  rights  reserved. 

Permission  to  make  digital  or  hard  copies  of  all  or  part  of  this  work  for 
personal  or  classroom  use  is  granted  without  fee  provided  that  copies  are 
not  made  or  distributed  for  profit  or  commercial  advantage  and  that  copies 
bear  this  notice  and  the  full  citation  on  the  first  page.  To  copy  otherwise,  to 
republish,  to  post  on  servers  or  to  redistribute  to  lists,  requires  prior  specific 
permission. 


A  General  Framework  for  Flow  Control  in  Wireless  Networks 


by 

Minghua  Chen 

B.Eng.  (Tsinghua  University)  1999 


A  dissertation  submitted  in  partial  satisfaction 
of  the  requirements  for  the  degree  of 

Doctor  of  Philosophy 
in 

Engineering-Electrical  Engineering  and  Computer  Sciences 

in  the 

GRADUATE  DIVISION 
of  the 

UNIVERSITY  OF  CALIFORNIA,  BERKELEY 


Committee  in  charge: 


Professor  Avideh  Zakhor,  Chair 
Professor  Scott  Shenker 
Professor  David  Aldous 


Fall  2006 


The  dissertation  of  Minghua  Chen  is  approved. 


Chair  Date 


Date 


Date 


University  of  California,  Berkeley 
Fall  2006 


A  General  Framework  for  Flow  Control  in  Wireless  Networks 


Copyright  ©  2006 


by 


Minghua  Chen 


Abstract 


A  General  Framework  for  Flow  Control  in  Wireless  Networks 


by 


Minghua  Chen 

Doctor  of  Philosophy  in  Engineering-Electrical  Engineering  and  Computer  Sciences 


University  of  California,  Berkeley 
Professor  Avideh  Zakhor,  Chair 


Flow  control,  including  congestion  control  for  data  transmission,  and  rate  control  for  mul¬ 
timedia  streaming,  is  an  important  issue  in  information  transmission  for  both  wired  and 
wireless  networks.  Proper  flow  control  allows  users  to  fairly  and  fully  utilize  available  band¬ 
width  without  the  possibility  of  congestion  collapse.  Failure  to  apply  proper  flow  control 
may  result  in  serious  performance  degradation  in  a  network. 

Although  the  problem  of  flow  control  has  been  successfully  addressed  in  wired  networks, 
it  is  still  open  in  wireless  networks.  Current  widely  accepted  solutions,  such  as  TCP,  assume 
that  congestion  is  the  only  cause  of  packet  loss,  and  as  such,  are  not  applicable  to  wireless 
networks  in  which  the  bulk  of  packet  loss  is  due  to  errors  at  the  physical  layer.  We  show 
that  this  often  results  in  bandwidth  underutilization  in  wireless  networks.  This  problem 
is  becoming  increasingly  more  serious  as  wireless  data  and  multimedia  services  are  being 
rapidly  deployed  commercially  on  carries  throughout  the  world  with  data  rates  of  up  to  one 
Mbps. 

In  this  thesis,  we  first  formulate  flow  control  in  wireless  networks  as  a  convex  optimiza¬ 
tion  problem.  We  then  propose  a  new  class  of  solutions  that  properly  adjust  the  number  of 
connections  of  a  user,  to  fully  utilize  wireless  bandwidth  and  minimize  end-to-end  packet 
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loss.  Our  solution  differs  from  all  existing  schemes  introduced  in  the  past  decade  in  the 
following  ways: 


•  First,  it  is  theoretically  guaranteed  to  be  optimal,  stable  and  scalable.  Practically,  in  a 
network  with  arbitrary  topology,  arbitrary  number  of  users,  and  arbitrary  initial  source 
rates,  our  proposed  schemes  guarantee  all  users’  source  rates  to  globally  exponentially 
converge  to  an  equilibrium.  This  convergence  guarantees  no  congestion  collapse  in 
the  network.  At  the  equilibrium,  all  bottlenecks  are  fully  utilized  and  users  are  fair 
to  each  other.  Furthermore,  proposed  schemes  are  fair  to  TCP/TFRC  protocols,  and 
are  therefore  amenable  to  incremental  deployment  in  the  current  Internet  where  TCP 
is  dominant. 

•  Second,  our  proposed  schemes  are  end-to-end  and  require  modifications  to  neither 
infrastructure  nor  transport  protocol  stack,  making  it  easy  to  deploy  in  practice. 

Based  on  this  approach,  we  have  designed  practical  schemes  for  data  transmission  over 
wireless  networks,  and  characterized  their  performance  using  simulations  and  actual  exper¬ 
iments  over  the  Verizon  Wireless  lxRTT  and  EVDO  CDMA  data  networks. 

This  work  implicitly  provides  a  general  framework  for  flow  control.  In  this  framework, 
both  users’  rates  and  the  number  of  connections  they  open  are  properly  controlled  to  pursue 
equilibrium  in  the  network.  We  show  it  is  sufficient  to  control  users’  rates  and  their  number 
of  connections  independently  in  two  separate  timescales,  in  order  to  guarantee  convergence 
to  the  desired  equilibrium.  This  two  timescale  approach  allows  modification  of  the  control 
law  in  one  timescale  without  affecting  the  one  in  the  other  timescale,  or  the  system’s  con¬ 
vergence.  This  framework  is  general  in  the  sense  that  its  usage  is  not  limited  to  the  problem 
we  study  in  this  thesis,  which  serves  as  an  ideal  platform  to  demonstrate  the  power  of  this 
approach. 


Professor  Avideh  Zakhor 
Dissertation  Committee  Chair 
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Chapter  1 


Introduction 


1.1  Motivation 

Flow  control,  including  congestion  control  for  data  transmission,  and  rate  control  for 
multimedia  streaming,  is  an  important  issue  in  information  transmission  for  both  wired 
and  wireless  networks.  Since  the  network  is  shared  among  all  users  and  the  connections 
are  stateless,  no  dedicated  bandwidth  is  allocated  to  any  end-to-end  connection.  Therefore, 
senders  and  receivers  should  have  a  mechanism  to  determine  the  sending  rate  at  which 
packets  are  injected  into  the  network. 

Flow  control  is  the  mechanism  to  distributively  determine  sending  rates  of  users  to 
achieve  the  following  goals:  (a)  full  utilization  of  bottleneck  links  by  ensuring  sending  rates 
are  not  too  low;  (b)  preventing  congestion  collapse  by  ensuring  sending  rates  are  not  too 
aggressive.  For  example  there  was  an  actual  network  collapse  of  the  Internet  in  Oct.  1986 
at  University  of  California  at  Berkeley  resulting  in  serious  performance  degradation  [25, 
Section  1],  Further,  it  would  be  ideal  for  sending  rates  to  converge  to  an  equilibrium,  such 
that  there  is  no  fluctuation  in  the  received  throughput,  resulting  in  constant  quality  in 
multimedia  streaming  applications.  Finally,  it  should  ensure  fairness  between  users  sharing 
common  links.  The  mechanism  is  required  to  be  distributive  in  the  sense  that  the  control 
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Table  1.1.  Flow  control  solutions  for  data  transmission  and  multimedia  streaming  in  wired 
and  wireless  networks. 


Wired  Network 

Wireless  Networks 

DATA 

TCP 

? 

MULTIMEDIA 

TFRC 

? 

must  be  done  based  only  on  local  information.  This  is  because  networks  are  growing  into 
arbitrarily  large  size,  and  any  centralized  solution  does  not  necessarily  scale. 

Transport  Control  Protocol  (TCP),  a  widely  accepted  congestion  control  protocol,  has 
been  extremely  successful  on  the  wired  Internet  since  its  first  implementation  by  Jacobson 
in  1988  [25].  TCP  Reno,  the  most  popular  TCP  version  today,  increases  its  window  size  by 
one  in  its  congestion  avoidance  stage  if  no  packet  is  lost  in  the  previous  round  trip  time, 
and  halves  the  window  size  otherwise.  Similarly,  TCP-Friendly  Rate  Control  (TFRC)  is 
a  rate  control  protocol  for  multimedia  streaming,  proposed  by  Floyd  et.  al.  [23].  There 
are  basically  three  advantages  to  rate  control  using  TFRC:  first,  it  does  not  cause  network 
instability,  thus  avoiding  congestion  collapse.  Second,  it  is  fair  to  TCP  flows,  which  is 
the  dominant  source  of  traffic  on  the  Internet.  Third,  the  TFRC’s  rate  fluctuation  is  lower 
than  TCP,  making  it  more  appropriate  for  multimedia  streaming  applications  which  require 
constant  quality.  The  key  assumption  behind  TCP  and  TFRC  is  that  packet  loss  is  a  sign  of 
congestion.  TCP  and  TFRC  have  been  shown  to  work  well  in  wired  networks.  As  a  result, 
every  computer  and  handheld  device  today  runs  TCP,  and  every  router  signals  congestion 
by  dropping  packets. 

In  wireless  networks  however,  packet  loss  can  also  be  caused  by  physical  channel  errors, 
thus  violating  this  assumption.  Neither  TFRC  nor  TCP  can  distinguish  between  packet  loss 
due  to  buffer  overflow  and  that  due  to  physical  channel  errors,  resulting  in  underutilization  of 
the  wireless  bandwidth.  Particularly,  our  experiments  over  Verizon  Wireless  lxRTT  wireless 
data  network  have  shown  that  TFRC  achieves  only  56%  of  the  available  wireless  bandwidth 
[16];  Balakrishnan  et.  al.  have  shown  that  TCP  Reno  achieves  only  22%  utilization  in 
wireless  LAN  environments  [10].  Hence  streaming  rate  control  and  congestion  control  over 
wireless  networks  are  still  open  issues,  as  indicated  in  Table  1.1  . 
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The  need  to  solve  the  problem  is  becoming  urgent  as  wireless  data  and  streaming  services 
are  becoming  increasingly  more  popular: 

•  Wireless  bandwidth  is  increasing.  For  example,  Verizon  wireless  EVDO  service  pro¬ 
vides  up  to  2  Mbps  bandwidth  for  a  single  user.  Such  high  bandwidth  can  support 
high  bit-rate  video  streaming  applications  with  quality  close  to  DVD  using  advanced 
video  coding  techniques  such  as  H.264  [27].  Such  high  bandwidth  also  implies  users 
can  download  music  files  on  the  order  of  seconds,  or  engage  in  instantaneous  online 
gaming. 

•  Handheld  wireless  devices  are  becoming  powerful.  For  example,  Lenovo  cell  phone 
model  ET560  has  an  Intel  X-Scale  400Mhz  processor  and  64MB  memory.  These 
powerful  handheld  devices  can  support  on-the-fly  processing  of  received  video  and 
music.  Hence  users  can  interact  with  multimedia  information  at  the  same  time  as 
they  are  being  received. 

•  Users  can  use  these  advanced  applications  for  up  to  hours  at  a  time,  thanks  to  the 
improved  battery  performance. 


1.2  Previous  Work  on  TCP/TFRC  over  Wireless 

There  have  been  a  number  of  efforts  to  improve  the  performance  of  TCP  or  TFRC  over 
wireless  [9-12,  14,  15,  18-20,  24,  33,  34,  42-45,  47,  48,  50,  51,  57,  58].  These  approaches 
either  hide  end-hosts  from  packet  loss  caused  by  wireless  channel  error,  or  provide  end- 
hosts  the  ability  to  distinguish  between  packet  loss  caused  by  congestion,  and  that  caused 
by  wireless  channel  error.  To  gain  a  better  understanding  of  the  spectrum  of  approaches  to 
rate  control  over  wireless,  we  briefly  review  TCP  and  TFRC  solutions  over  wireless;  we  will 
provide  a  fundamental  overview  of  all  these  solutions  in  Section  3.2. 

Snoop,  a  well-known  solution,  is  a  TCP- AWARE  local  retransmission  link  layer  approach 
[10].  A  Snoop  module  resides  on  router  or  base  station  on  the  last  hop,  which  is  assumed 
to  be  wireless,  and  records  a  copy  of  every  forwarded  packet.  Assuming  snoop  module  can 
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access  TCP  acknowledgement  packets  (ACK)  from  the  TCP  receiver,  it  looks  into  the  ACK 
packets  and  carries  out  local  retransmissions  when  a  packet  is  identified  to  be  corrupted  by 
wireless  channel  errors.  While  doing  the  local  retransmission,  the  ACK  packet  is  suppressed 
and  not  forwarded  to  the  TCP  sender.  Other  similar  approaches  based  on  local  link  layer 
retransmission  include  [15,  18,  19,  24,  43,  44],  These  schemes  can  potentially  be  extended 
to  TFRC  in  order  to  improve  performance,  by  using  more  complicated  treatment  of  ACK 
packets  from  TFRC  receiver. 

Explicit  Loss  Notification  (ELN)  can  also  be  applied  to  notify  TCP/TFRC  sender  when 
packet  loss  is  caused  by  wireless  channel  errors  rather  than  congestion  [9,  58] .  In  this  case, 
TFRC  can  take  into  account  only  the  packet  loss  caused  by  congestion  when  adjusting  the 
streaming  rate. 

End-to-end  statistics  can  be  used  to  detect  congestion  when  a  packet  is  lost  [11,  12, 
14,  20,  33,  34,  42,  45,  47,  48,  51,  57].  For  example,  by  examining  trends  in  the  one-way 
delay  variation,  Parsa  and  Garcia-Luna-Aceves  [20]  interpret  loss  as  a  sign  of  congestion  if 
one-way  delays  are  increasing,  and  a  sign  of  wireless  channel  error  otherwise.  One-way  delay 
can  be  associated  with  congestion  in  the  sense  that  it  monotonically  increases  if  congestion 
occurs  as  a  result  of  increased  queueing  delay,  and  remains  constant  otherwise.  Similarly, 
Barman  and  Matta  have  proposed  a  loss  differentiation  scheme  based  on  the  assumption 
that  the  variance  of  round  trip  time  is  high  when  congestion  occurs,  and  is  low  otherwise 
[11]- 

Cen  et.  al.  present  an  end-to-end  based  approach  to  facilitate  streaming  over  wireless 
[14].  They  combine  packet  inter-arrival  interval  and  relative  one  way  delay  to  differentiate 
between  packet  loss  caused  by  congestion  and  that  due  to  wireless  channel  errors.  There 
are  two  key  observations  behind  their  approach;  first,  relative  one  way  delay  increases 
monotonically  if  there  is  congestion;  second,  inter-arrival  interval  is  expected  to  increase  if 
there  is  packet  loss  caused  by  wireless  channel  errors.  Therefore,  these  two  statistics  can 
differentiate  between  congestion  and  wireless  errors.  Nevertheless,  the  high  wireless  error 
misclassification  rate  may  result  in  underutilizing  the  wireless  bandwidth,  as  shown  in  [14] . 
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Yang  et.  al.  [57]  also  propose  a  similar  approach  to  improve  video  streaming  performance 
in  presence  of  wireless  error,  under  the  assumption  that  wireless  link  is  the  bottleneck. 

Other  schemes  such  as  [12,  33,  45,  47,  48,  51]  that  use  end-to-end  statistics  to  detect 
congestion,  can  also  be  combined  with  TFRC  for  rate  control.  The  congestion  detection 
scheme  can  be  used  to  determine  whether  or  not  an  observed  packet  loss  is  caused  by 
congestion;  TFRC  can  then  take  into  account  only  those  packet  loss  caused  by  congestion 
when  adjusting  streaming  rate. 

Tang  et.  al.  proposed  the  idea  of  using  small  dummy  packets  to  actively  probe  whether 
the  network  is  congested  in  case  of  packet  loss,  so  as  to  differentiate  between  packet  loss 
due  to  congestion  and  that  due  to  channel  error  [50].  Yang  et.  al.  [55]  propose  a  cross-layer 
scheme  that  uses  link  layer  information  to  determine  whether  a  packet  loss  is  caused  by 
channel  error  or  congestion,  assuming  that  only  the  last  link  is  wireless.  In  this  approach, 
when  a  packet  is  lost,  TFRC  goes  beyond  layering  abstraction  and  enquiries  the  link  layer 
about  the  recent  signal  strength.  The  packet  loss  is  recognized  to  be  a  result  of  wireless 
channel  error  if  recent  signal  strength  is  low,  and  due  to  congestion  otherwise. 

The  disadvantage  of  end-to-end  statistics  based  approaches  is  that  congestion  detection 
schemes  based  on  statistics  are  not  sufficiently  accurate,  and  they  either  require  cross  layer 
information  or  modifications  to  the  transport  protocol  stack. 

Another  alternative  is  to  use  non-loss  based  rate  control  schemes.  For  instance,  TCP 
Vegas  [13],  in  its  congestion  avoidance  stage,  uses  queueing  delay  as  a  measure  of  congestion, 
and  hence  could  be  designed  not  to  be  sensitive  to  any  kind  of  packet  loss,  including  that 
due  to  wireless  channel  error.  It  is  also  possible  to  enable  the  routers  with  ECN  marking 
capability  to  do  rate  control  using  ECN  as  the  measure  of  congestion[21].  As  packet  loss 
no  longer  corresponds  to  congestion,  ECN  based  rate  control  does  not  adjust  sending  rate 
upon  observing  a  packet  loss. 

In  summary,  although  flow  control  in  wired  networks  has  been  successfully  addressed, 
extending  the  known  solutions,  i.e.  TCP  and  TFRC,  to  wireless  scenarios  is  not  trivial. 
All  existing  solutions  either  require  support  from  network  infrastructure  such  as  routers 
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and  base  stations,  or  require  modifications  to  the  transport  layer  protocol  stack,  in  the 
operating  systems  of  every  computer  and  handheld  device.  Infrastructure  providers  such 
as  Cisco,  and  operating  system  providers  such  as  Microsoft  are  reluctant  to  carry  out  such 
massive  modifications  to  their  products. 

Therefore,  an  open  question  of  practical  interest  is  the  following: 

Is  it  possible  to  solve  the  problem  of  flow  control  in  wireless  networks ,  without  changing 
today’s  network  infrastructure,  operating  systems,  protocol  stack,  or  violating  the  end-to- 
end  principle? 


1.3  Previous  Work  on  Flow  Control 

In  the  past  eight  years,  there  has  been  a  great  deal  of  theoretical  research  on  under¬ 
standing  and  designing  distributed  end-to-end  network  flow  control  algorithms.  A  widely 
recognized  setting  has  been  introduced  by  Kelly  et.  al.  in  the  seminal  work  [29],  and  is 
based  on  a  fluid- flow  approximation  of  packets  propagating  over  links;  it  associates  a  util¬ 
ity  function  to  each  flow,  a  cost  function  to  each  resource,  and  maximizes  the  aggregate 
net  system  utility  function.  Under  this  framework,  flow  control  schemes  can  be  viewed  as 
algorithms  to  compute  the  optimal  solution  to  this  maximization  problem.  Kelly  and  his 
colleagues  have  proposed  two  complementary  flow  control  algorithms,  the  primal  and  the 
dual  [29]. 

In  primal  algorithms,  the  users  adapt  their  sending  rates  dynamically  based  on  the 
costs  that  incur  along  the  path  through  the  network,  while  the  routers  determine  their 
prices  directly  from  the  arrival  rates  at  the  links  according  to  a  static  law.  Hence  the 
primal  algorithms  are  end-to-end,  with  only  simple  feedback  prices  from  the  network.  One 
example  of  pricing  strategy,  which  is  used  in  TCP  Reno,  exploits  the  packet  loss  rate. 
The  algorithms  analyzed  by  Kunniyur  and  Srikant  [31],  Alpcan  and  Basar  [7],  Vinnicombe 
[53]  belong  to  this  class.  The  stability  of  the  various  primal  algorithms  is  investigated  in 
[8,  26,  29,  53]. 
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In  dual  algorithms,  on  the  other  hand,  the  routers  adapt  the  prices  dynamically  based 
on  the  link  rates,  and  the  users  select  a  static  law  to  determine  the  source  rates  directly 
from  the  prices  along  the  path  and  the  source  parameters.  These  schemes  rely  on  the 
network  resources  to  implement  the  congestion  control.  The  algorithms  proposed  by  Low 
and  Lapsley  [36],  Paganini  et.  al.  [41],  Yaiche  et.  al.  [54]  belong  to  this  class.  The  stability 
properties  of  various  dual  algorithms  are  investigated  in  [29,  41]. 

There  is  also  another  class  of  algorithm  named  primal-dual,  where  both  the  prices  and 
the  source  rates  are  updated  dynamically  by  routers  and  users,  respectively.  The  algorithms 
proposed  by  Low  and  Lapsley  [36],  Kunniyur  and  Srikant  [28],  Paganini  et.  al  [41]  belong 
to  this  class.  The  stability  of  various  primal-dual  algorithms  is  investigated  in  [28,  36,  41]. 

All  these  frameworks  can  be  used  to  understand  and  design  the  congestion  control 
algorithms  and  predict  their  performance.  For  example,  Low  has  pointed  out  that  different 
versions  of  TCP,  as  well  as  queue  management  algorithms  such  as  DropTail  and  RED,  can 
be  analyzed  under  the  same  duality  model  with  different  utility  and  update  functions  [35]. 
Kunniyur  and  Srikant  have  investigated  two  algorithms  that  can  be  used  to  understand  the 
behavior  of  TCP  [31].  Kelly  has  shown  TCP  to  be  a  primal  like  algorithm  with  packet  loss 
rate  as  the  associated  price  function  [28] ;  in  this  work  the  stability  for  the  system  with  and 
without  delay,  with  and  without  disturbance,  are  reviewed  and  further  investigated.  Kelly’s 
paper  also  discusses  the  selection  of  the  TCP  parameters,  in  order  to  achieve  a  scalable 
robust  congestion  control. 

Specifically,  networks  consisting  of  TCP  Reno  and  routers  implementing  DropTail  or 
RED,  have  shown  to  achieve  all  flow  control  goals: 

•  Users  adjust  sending  rate  based  on  only  the  end-to-end  packet  loss  observed,  and 
routers  drop  packets  only  based  on  the  difference  between  aggregate  incoming  rate 
and  links’  capacities.  Hence,  all  the  algorithms  can  be  implemented  in  a  distributed 
manner. 

•  It  has  been  shown  that  sending  rates  controlled  by  TCP  Reno  converge  to  an  optimal 
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and  unique  equilibrium  exponentially  fast  [32].  Optimality  of  the  equilibrium  lies  in 
the  fact  it  maximizes  an  aggregate  net  utility  [35,  38]. 

•  At  the  converged  equilibrium,  all  bottleneck  links  are  fully  utilized,  and  users  are  fair 
to  each  other  under  certain  fairness  criteria  [38]. 

Nevertheless,  all  these  frameworks  either  work  for  wired  networks  only,  or  require  addi¬ 
tional  functionalities  from  infrastructures,  or  require  change  to  the  existing  transport  layer 
protocol  stack.  These  either  limit  their  applicability  to  wireless  networks,  or  would  make 
them  hard  to  be  deployed  in  practice. 

Therefore,  an  open  question  of  theoretical  interest  is  the  following: 

What  is  the  framework  for  flow  control  problem  in  wired  or  wireless  networks,  without 
modifying  today’s  network  infrastructure,  operating  systems,  or  the  protocol  stacks? 


1.4  Thesis  Organization 

In  this  thesis,  we  aim  at  providing  answers  to  the  two  open  questions  mentioned  in 
the  previous  sections.  We  assume  a  wireless  link  is  associated  with  a  fixed  bandwidth  and 
a  fixed  packet  loss  rate  caused  by  the  physical  channel  errors.  We  first  show  that  in  the 
presence  of  the  packet  loss  caused  by  physical  channel  error,  flow  control  in  the  wireless 
network  can  be  formulated  as  the  same  concave  optimization  problem  defined  by  Kelly  in 
wired  networks [28].  TCP  and  TFRC  in  the  wireless  networks  pursue  the  optimal  solution 
using  inaccurate  feedback1.  All  existing  approaches  to  this  problem  correct  the  inaccurate 
feedback  by  casting  modifications  to  existing  protocols,  such  as  TCP,  or  infrastructure 
elements  such  as  routers,  thereby  making  them  hard  to  deploy  in  practice. 

In  this  thesis,  we  formulate  the  problem  as  another  concave  optimization  problem  with 
a  different  utility  function,  followed  by  a  new  class  of  solutions.  Our  approach  is  end-to- 
end,  and  achieves  reasonable  performance  by  adjusting  the  number  of  users’  connections 
according  to  a  properly  selected  control  law.  The  control  law  is  based  on  only  one  bit  of 


Tn  [6,  17],  we  have  also  shown  the  similar  formulation  using  the  framework  in  [29]. 


information,  which  can  be  reliably  measured  at  the  application  layer.  We  show  that  the 
resulting  control  system  has  a  unique  stable  equilibrium  that  solves  the  concave  optimization 
problem,  implying  scalability  and  optimality  of  the  solution.  We  then  apply  our  results  to 
design  a  practical  rate  control  scheme  for  data  transmission  over  wireless  networks,  and 
characterize  its  performance  using  NS-2  simulations,  and  actual  experiments  over  Verizon 
Wireless  lxRTT  and  EVDO  CDMA  data  network.  Analysis  and  simulation  results  indicate 
our  scheme  is  applicable  to  both  wired  and  wireless  scenarios. 

This  thesis  is  organized  as  follows.  In  Chapter  2,  we  discuss  the  available  design  space, 
and  study  a  simple  case  of  streaming  over  one  wireless  link  in  order  to  gain  intuition  about 
the  problem.  Problem  formulation  and  analysis  are  included  in  Chapter  3.  A  new  approach 
addressing  the  problem  is  proposed  in  Chapter  3,  together  with  analysis  for  its  optimality 
and  stability.  Chapter  4  shows  the  design  of  practical  schemes  following  the  insights  derived 
from  theoretical  analysis  of  Chapter  3.  Trade-off  analysis  between  bandwidth  utilizations 
and  responsiveness  over  a  single  user,  single  wireless  link  scenario  are  also  included  in 
Chapter  4.  NS-2  simulations  and  actual  experiments  over  lxRTT  and  EVDO  CDMA  data 
networks  are  included  in  Chapter  5.  Chapter  6  concludes  the  thesis  with  discussions  and 
future  work. 
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Chapter  2 


Design  Space  and  A  Simple  Case 
Study 


In  this  chapter,  we  first  argue  that  our  design  space  is  the  application  layer,  since 
modifications  to  underlying  layers  are  overly  restrictive  in  our  setting.  We  study  a  simple 
case  of  streaming  over  one  wireless  link  in  order  to  gain  intuition  on  how  modifications  to 
application  layer  can  improve  wireless  bottleneck  utilization. 


2.1  Design  Space 

In  previous  chapter,  we  stated  our  objective  of  not  requiring  modifications  to  network 
infrastructure  such  as  routers,  or  transport  protocol  stack  such  as  TCP.  As  seen  from  Figure 
2.1,  this  implies  keeping  IP  and  underlying  physical  layers  intact.  Also  no  modification  to 
transport  protocol  stack  implies  intact  transport  layer.  Therefore,  the  only  available  design 
space  is  the  application  layer. 

In  the  next  section,  we  will  intuitively  show  what  can  be  done  in  application  layer  to 
address  the  problem  of  flow  control  in  wireless  networks. 
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Figure  2.1.  Layering  abstraction  of  packets  traveling  across  network.  Under  the  requirement 
of  no  modification  to  infrastructure  and  transport  layer  protocol,  the  only  available  design 
space  lies  in  application  layer. 


2.2  A  Simple  Case  and  Resulting  Intuition 


A  simple  scenario  for  data  transmission  and  streaming  over  one  wireless  link  is  shown 
in  Figure  2.2  where  a  server  s  in  the  wired  network  is  sending  data  or  streaming  video 
to  a  receiver  r  in  the  wireless  network.  The  wireless  link  is  assumed  to  have  available 
bandwidth  Bw,  and  packet  loss  rate  pw,  caused  by  wireless  channel  error.  There  could  also 
be  packet  loss  caused  by  congestion  at  node  2,  denoted  by  pc.  The  end-to-end  packet  loss 
rate  observed  by  receiver  is  denoted  by  p,  and  the  streaming  rate  is  denoted  by  T.  We 
refer  to  the  wireless  channel  as  underutilized  if  the  streaming  throughput  is  less  than  the 
maximum  possible  throughput  over  the  wireless  link,  i.e.  T(1  —  p)  <  BW{1  —  pw). 


video  □  □ : 


wired  links 


wireless  link 


(Bw>Pw) 


Figure  2.2.  Typical  scenario  for  streaming  over  wireless. 

Given  this  scenario,  we  assume  the  following.  First,  there  are  no  cross  traffic  at  either 
node  1  or  node  2.  Second,  in  the  long  term,  the  wireless  link  is  assumed  to  be  the  bottleneck. 
By  this,  we  mean  there  is  no  congestion  at  node  1.  Third,  we  assume  there  is  no  congestion 
and  queuing  delay  at  node  2  if  and  only  if  wireless  bandwidth  is  underutilized,  i.e.  we 
achieve  pc  =  0  and  minimum  round  trip  time,  defined  as  RTTmin ,  if  and  only  if  T  <  Bw. 
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When  T  >  Bw,  we  have  pc  >  0  and  rtt  >  RTTmin.  Fourth,  Bw  and  pw  are  assumed  to 
be  constant,  at  least  on  the  timescale  analysis  is  carried  on;  packet  loss  caused  by  wireless 
channel  error  is  assumed  to  be  random  and  stationary.  Fourth,  for  simplicity,  the  backward 
route  is  assumed  to  be  error-free  and  congestion-free. 

As  pointed  out  in  literature,  the  problem  of  TCP/TFRC  over  wireless  is  underutilization 
[10,  16].  We  now  investigate  the  sufficient  and  necessary  condition  for  underutilization  for 
this  very  simple  scenario  to  gain  intuition  on  how  to  solve  it. 


2.2.1  A  Sufficient  and  Necessary  Condition  for  Underutilization 


We  use  the  following  model  for  TCP/TFRC  sending  rate  in  the  analysis[22]: 

kS 

T  = 


(2.1) 


rtty/p 

where  T  represents  the  sending  rate,  S  is  the  packet  size,  rtt  is  the  end-to-end  round 
trip  time,  p  is  the  end-to-end  packet  loss  rate,  and  A:  is  a  constant  factor.  Although  this 
model  has  been  refined  to  improve  accuracy  [23,  40],  it  is  simple,  easy  to  analyze,  and  more 
importantly,  captures  all  the  fundamental  factors  that  affect  the  sending  rate.  Furthermore, 
the  results  we  derive  based  on  this  simple  model  can  be  extended  to  other  more  sophisticated 
models,  such  as  the  one  used  in  [23]. 

The  overall  packet  loss  rate  is  p.  a  combination  of  pw  and  pc,  and  can  be  written  as: 


P  =  Pw  +  (1  -  Pw)Pc- 


(2.2) 


This  shows  that  pw  is  a  lower  bound  for  p.  and  that  the  bound  is  reached  if  and  only  if 
there  is  no  congestion,  i.e.  pc  =  0.  Combining  this  observation  and  Equation  (2.1),  an 
upper  bound,  Tb,  on  the  streaming  rate  of  one  TFRC  connection  can  be  derived  as  follows: 


T  < 


kS 


RT  Tmin  \jpw 


=  Tb 


(2.3) 


If  there  is  no  congestion,  i.e.  pc  =  0,  and  hence  no  queuing  delay  caused  by  congestion,  we 
get  rtt  =  RTTmin,  p  =  pw,  and  T  achieves  the  upper  bound  T  =  Tb  in  Equation  (2.3).  In 
this  case,  the  throughput  is  Tb(  1  —  pw),  which  is  the  upper  bound  of  throughput  given  one 
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TFRC  connection  for  the  scenario  shown  in  Figure  2.2.  Based  on  these,  we  can  state  the 
following  theorem: 

Theorem  2.2.1.  Given  the  above  scenario  and  assumptions,  a  sufficient  and  necessary 
condition  for  one  TFRC  connection  to  under-utilize  wireless  link  is 

Tb  <  Bw.  (2.4) 

Proof.  Since  Tb(  1  —  pw )  is  the  upper  bound  of  one  TFRC’s  throughput,  clearly  Equation 
(2.4)  implies  under-utilization  of  the  wireless  channel,  and  hence  the  ” sufficient”  part  of 
the  Theorem  is  obvious.  To  see  the  necessary  part,  note  that  if  under-utilization  happens, 
i.e.  T(1  —  p)  <  Bw(l  —  pw),  then  no  congestion  happens,  thus  rtt  =  RTTmin ,  p  =  pw  and 
T  =  Tb,  resulting  in  Tb(  1  -  p)  <  Bw(  1  -pw).  □ 

Theorem  2.2.1  implies  that  if  the  available  bandwidth  is  larger  than  the  highest  sending 
rate  one  TCP/TFRC  can  achieve,  then  underutilization  happens.  In  essence,  the  approaches 
taken  in  [9,  10,  12,  14,  15,  18-20,  24,  33,  34,  42-45,  47,  48,  51,  57,  58]  ensure  the  condition  in 
Equation  (2.4)  is  not  satisfied,  through  modifications  to  network  infrastructure  or  protocols. 
For  example  in  the  TFRC- AWARE  Snoop-like  solution,  pw  becomes  effectively  zero  after 
local  retransmissions,  and  thus  Equation  (2.4)  can  never  be  satisfied.  By  effectively  setting 
pw  =  0,  Snoop-like  module  translates  the  new  problem,  i.e.  rate  control  for  streaming  over 
wireless,  into  an  old  one,  i.e.  rate  control  for  streaming  over  wired  networks,  for  which  a 
known  solution  exists.  Similar  observations  can  be  made  for  other  solutions  such  as  the 
end-to-end  statistics  based  approaches  [12,  14,  20,  33,  34,  42,  45,  47,  48,  51,  57].  Similarly, 
ELN  and  end-to-end  statistics  based  approaches  make  TFRC  not  respond  to  packet  loss 
caused  by  wireless  channel  errors,  thus  not  taking  pw  into  account  when  adjusting  streaming 
rate.  This  is  effectively  the  same  as  setting  pw  =  0,  thus  improving  the  performance  of  the 
TFRC  connection. 

Theorem  2.2.1  also  indicates  the  two  regions  in  which  TCP  operates,  as  shown  in  Figure 
2.3.  In  the  wired  scenario,  TCP  keeps  increasing  its  rate  until  the  rate  hits  wired  link 
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Figure  2.3.  Sending  rate  of  TCP:  limited  by  link  capacity  in  wired  scenario  (left);  limited 
by  wireless  channel  error  in  the  wireless  scenario  (right). 


capacity  and  packet  loss  due  to  router  queue  overflow  is  observed.  Therefore,  TCP’s  rate 
is  limited  by  the  link  capacity,  operating  in  capacity-limited  region.  In  wireless  scenario, 
however,  TCP  can  not  differentiate  between  packet  loss  caused  by  congestions  and  that 
due  to  physical  layer  errors.  Consequently,  TCP  halves  its  rate  when  packet  loss  caused  by 
wireless  channel  error  is  observed,  which  might  happen  far  before  TCP’s  rate  reaches  the 
link  capacity.  Therefore,  in  channel-error-limited  region,  TCP’s  rate  is  limited  by  wireless 
channel  error,  resulting  in  underutilization.  On  the  other  hand,  TCP  achieves  full  utilization 
if  it  operates  in  the  capacity-limited  region.  Similar  analysis  and  intuitions  apply  to  TFRC 
as  well. 


2.2.2  A  Strategy  to  Reach  the  Optimal  Performance 

It  is  not  necessary  to  avoid  the  condition  in  Equation  (2.4)  in  order  to  achieve  full 
utilization  for  one  application.  This  is  because  it  is  conceivable  to  use  multiple  simultaneous 
connections  for  a  given  streaming  application.  The  total  throughput  of  the  application  is 
expected  to  increase  with  the  number  of  connections  until  it  reaches  the  hard  limit  of 
Bw{  1  —  pw).  Intuitively,  although  each  single  TCP/TFRC  still  operates  in  channel-error- 
limited  region,  the  aggregate  rates  of  multiple  parallel  TCP/TFRC  can  achieve  the  link 
capacity,  and  hence  improve  utilization. 
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Analysis  on  the  Optimal  Number  of  Connections 


Given  the  scenario  shown  in  Figure  2.2,  and  the  associated  assumptions,  we  now  argue 
that  multiple  connections  can  be  used  to  achieve  optimal  performance,  i.e.  throughput  of 
Bw(  1  —  pw),  and  packet  loss  rate  of  pw.  To  see  this,  let  us  consider  a  simple  example  in 
which 

2  5  kS 

Bw{  1  -  Pw)  =  - t=(1  ~Pw)  =  2.5Tb(l  -pw) 

m  J-  min  yj Pw 

By  opening  one  TCP/TFRC  connection  with  packet  size  S,  the  application  achieves  a 
throughput  of  t==(1  —  pw)  =  2/(1  —  pw )  and  packet  loss  rate  of  pw.  This  is  because 

fi'-L  J-  miny/Pw  v  7  v  7 

according  to  Theorem  2.2.1,  underutilization  implies  rtt  =  RTTmin,  p  =  pw  and  T  = 

_ kS_ _  _  rp 

RTTmin  \J Pw  ^ 

Let  us  now  consider  the  case  with  two  TCP/TFRC  connections  from  sender  s  to  receiver 
r  in  Figure  2.2.  It  is  easy  to  see  that  pw  for  each  of  the  two  TCP/TFRC  connections  remain 
unchanged  from  the  case  with  one  TCP/TFRC  connection.  Thus  the  throughput  upper 
bound  for  each  of  the  two  TCP/TFRC  connections  is  n ;  ;  ^  (I  —pw)  =  1/(1  —pw)i  and 

the  aggregate  throughput  upper  bound  for  both  of  them  is  2  kS  7 —  ( 1  —pw )  =  22/(1—  pw), 

■K'-L  ^  min  y  Pw  v  7  v  7 

which  is  smaller  than  Bw(l  —  pw),  implying  channel  under-utilization.  Hence  rtt  =  RTTm,in , 
pc  =  0,  and  thus  p  =  pw.  The  throughput  for  each  connection  is  then  ^  ^(1  —  pw). 

1  minyPw  v  7 

Consequently,  the  total  throughput  for  both  connections  is  2  RTTK  /— (1  —pw)  with  packet 

BiR  1  miny/Pw  v  7 

loss  rate  at  pw. 

A  similar  argument  can  be  repeated  with  three  TCP/TFRC  connections,  except  that 
the  wireless  channel  is  no  longer  under- utilized  and  rtt  >  RTTmin.  Furthermore,  if  the 
buffer  on  node  2  overflows  then  pc  will  no  longer  be  zero  and  hence  using  Equation  (2.2)  we 
get  p  >  pw.  In  this  case  the  wireless  link  is  still  fully  utilized  at  T(1  —  p)  =  Bw(  1  —  pw),  but 
round  trip  time  is  no  longer  at  the  minimum  value  RTTmin ,  and  overall  packet  loss  rate  p 
could  exceed  pw,  i.e.  the  overall  packet  loss  rate  in  the  two  connections  case. 

In  general,  given  Bw,  pw,  and  the  packet  size  S  for  each  connection,  it  can  be  shown 
that  when  full  wireless  channel  utilization  occurs,  the  optimal  number  of  connections,  nopt, 


15 


satisfies: 


BllliX  Pw )  —  n0pt  Tjrprp  , - (1  Pw) 

*  J-  min\/Pw 

o  d  RTTminy/Pw  ,n 

=*>  nopt<S  =  ^ -  (2.5) 

Thus  what  really  matters  is  the  product  of  nopt  and  S,  and  it  is  always  possible  to  achieve 
full  wireless  channel  utilization  by  choosing  nopt  to  be  an  integer,  and  by  selecting  S  ac¬ 
cordingly1.  It  is  also  possible  to  analyze  the  case  with  different  packet  sizes  for  different 
connections,  but  this  is  harder  to  do,  and  it  is  not  fundamentally  different  from  the  case 
with  the  same  packet  size  for  all  connections.  For  the  case  with  the  fixed  packet  size  at  S, 
the  optimal  number  of  connections  is  given  by 

=  f^opt  (2.6) 

resulting  in  throughput  of  hopt  RTTkb  ^-(1  —  pw)  and  packet  loss  rate  of  pw. 

To  show  that  opening  more  than  nopt  connections  results  in  larger  rtt,  or  possibly  higher 
end-to-end  packet  loss  rate,  assume  nopt  and  S  lead  to  the  optimal  performance,  and  consider 
opening  nopt+5n  connections,  where  5n  is  a  positive  integer.  Denoting  the  end-to-end  packet 
loss  rate  as  p'  for  this  case,  the  overall  throughput  is  given  by  ( nopt  +  Sri) 

Bw{  1  —  pw)  and  hence 

{nopt  +  Sn)S  =  Bw  - — (2.7) 

1  —  p  k 

Comparing  the  above  equation  with  Equation  (2.5),  and  taking  into  account  that  the 
right  hand  sides  of  Equations  (2.5)  and  (2.7)  are  monotonically  increasing  functions  with 
respect  to  overall  packet  loss  rate  and  round  trip  time,  we  conclude  that  either  rtt  >  RTTmin 
and/or  p'  >  pw. 

The  intuition  here  is  that  as  number  of  connections  exceeds  nopt ,  the  sending  rate  of 
each  connection  has  to  decrease.  Thus  by  Equation  (2.1),  the  product  rttyjp  has  to  increase, 
so  either  rtt  increases  or  p  increases,  or  they  both  increase.  In  practice,  as  the  number  of 
connections  exceeds  nopt,  initially  p  remains  constant  and  rtt  increases  due  to  the  increase  on 

xOf  course  pw  may  also  change  when  packet  size  changes,  but  for  the  sake  of  simplicity,  we  assume  pw  is 
stable  as  packet  size  changes.  Analysis  can  be  extended  given  a  relation  between  pw  and  S.  The  point  here 
is  to  change  packet  size  to  achieve  finer  granularity  in  increase/decrease. 


B,„ 


RTTminy/P- w 
kS 
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queueing  delay  at  node  2,  i.e.  rtt  >  RTTmin;  if  the  number  of  connections  keeps  increasing 
and  buffer  on  node  2  overflows,  rtt  will  then  stop  increasing,  and  p  begins  to  increase. 
Eventually  we  get  both  rtt  >  RTTmin  and  p  >  pw. 

To  summarize,  if  there  are  too  few  TCP/TFRC  connections  so  that  the  aggregate 
throughput  is  smaller  than  Bw(  1  —  pw),  wireless  channel  becomes  under- utilized.  If  the 
number  of  connections  is  chosen  optimally  based  on  Equation  (2.5),  then  wireless  channel 
becomes  fully  utilized,  the  total  throughput  becomes  Bw(  1  — pw ),  the  rtt  =  RTTmin ,  and 
the  overall  packet  loss  rate  is  at  the  lower  bound  pw.  However,  if  the  number  of  connections 
exceeds  nopt,  even  though  the  wireless  channel  continues  to  be  fully  utilized  at  Bw(  1  —  pw), 
the  rtt  will  increase  beyond  RTTmin  and  later  on  packet  loss  rate  can  exceed  the  lower 
bound  pw. 

Simulations  and  Experimental  Verification 

To  validate  the  above  conclusions,  we  carry  out  both  NS-2  [3]  simulations  and  actual 
experiments  over  Verizon  Wireless  lxRTT  CDMA  data  network.  The  topology  for  NS-2 
simulations  is  the  same  as  the  one  shown  in  Figure  2.2  with  the  following  settings:  Bw  = 
1  Mbps,  RTTm,in  =  168  ms,  S  =  1000  bytes,  and  pw  varying  from  0.0  to  0.16.  Also,  no 
cross  traffic  is  introduced  for  illustration  purposes.  Within  NS-2,  we  stream  1,  2,  4,  8,  16 
and  32  TFRC  connections  from  a  fixed  host  to  mobile  hosts  for  1000  seconds.  The  wireless 
link  is  modeled  as  a  wired  link  with  an  exponential  random  packet  loss  model. 

The  results  of  NS-2  simulations  indicating  throughput,  packet  loss  rate  and  round  trip 
time  as  a  function  of  wireless  channel  error  rate,  pw,  for  different  number  of  connections, 
are  shown  in  Figure  2.4.  There  are  three  observations  to  be  made.  First,  for  a  given  pw , 
throughput  increases  with  the  number  of  connections  up  to  a  point,  after  which  there  is 
a  saturation  effect.  For  example,  for  pw  =  0.04  we  need  to  open  at  least  4  connections  to 
maximize  the  throughput.  Second,  for  a  fixed  pw,  opening  too  many  connections  results  in 
either  higher  packet  loss  rate,  or  higher  round  trip  time  than  RTTmin,  or  both;  for  instance, 
as  seen  from  Figure  2.4,  at  pw  =  0.04,  opening  8  connections  results  in  increase  in  round 
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trip  time  but  not  in  packet  loss  rate;  however,  opening  16  or  32  connections  results  in  packet 


loss  rate  to  be  higher  than  0.04,  and  larger  round  trip  time.  Third,  given  Bw,  pw,  RTTm.in , 
and  S,  there  is  an  ”  optimal”  number  of  connections  with  the  highest  throughput  and  the 
lowest  packet  loss  rate;  for  example,  for  pw  =  0.04,  the  optimal  number  of  connections  is 
around  4  or  5. 

Similar  experiments  are  carried  out  on  Verizon  Wireless  lxRTT  CDMA  data  network. 
The  lxRTT  CDMA  data  network  is  advertised  to  operate  at  data  speeds  of  up  to  144  kbps 
for  one  user.  As  we  explore  the  available  bandwidth  for  one  user  using  UDP  flooding,  we 
find  the  highest  average  available  bandwidth  averaged  over  30  minutes  to  be  between  80 
kbps  and  97  kbps.  In  our  experiments,  we  stream  for  30  minutes  from  a  desktop  on  wired 
network  in  EECS  department  at  U.C.  Berkeley  to  a  laptop  connected  via  lxRTT  CDMA 
modem  using  1,  2  and  3  connections  with  packet  size  of  S  =  1460  bytes.  We  measure 
the  total  throughput,  packet  loss  rate  and  round  trip  time  as  shown  in  Table  2.1.  Clearly, 
the  optimal  number  of  connections  is  2.  Specifically,  the  loss  rate  is  slightly  higher  for  3 
connections  than  for  2,  while  the  throughput  is  more  or  less  the  same  for  2  and  3  connections. 

Table  2.1.  Experimental  results  for  Verizon  Wireless  lxRTT  CDMA  data  network. 


number  of 

conn.’s 

throughput 

(kbps) 

rtt 

(ms) 

pkt  loss 
rate 

one 

57 

1357 

0.018 

two 

48.2+45.6=94 

2951 

0.032 

three 

33.2+31.9+27.8=93 

2863 

0.046 

Based  on  the  above  analysis  and  experiments  on  the  simple  case,  one  intuitive  strategy 
leading  to  good  performance  can  be  described  as  follows: 

Keep  increasing  the  number  of  connections  until  an  additional  connection  results  in  increase 
of  end-to-end  round  trip  time  or  packet  loss  rate. 

In  this  chapter,  we  have  clarified  the  available  design  space,  and  have  developed  insights 
on  how  to  address  the  underutilization  problem  of  flow  control  in  wireless,  by  only  adjusting 
the  number  of  parallel  TCP/TFRC  connections.  Although  the  analysis  is  based  on  a  simple 
scenario  and  is  very  limited,  the  insights  we  gain  are  in  fact  is  general,  and  can  be  extended 
to  arbitrary  scenarios.  In  the  next  chapter,  we  formulate  the  problem  rigorously  and  derive 
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Figure  2.4.  NS-2  simulations  of  the  simple  case  study:  (a)  End-to-end  throughput,  (b) 
packet  loss  rate,  and  (c)  round  trip  time  as  a  function  of  wireless  channel  error  rate,  pWj  for 
different  number  of  connections. 
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solutions,  which  achieve  all  of  desired  goals  and  provide  performance  guarantees.  We  will 
see  the  proposed  solution  shares  the  same  basic  characteristics  we  have  shown  in  the  current 
chapter. 
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Chapter  3 


Problem  Formulation  and 
Proposed  Solution 


In  this  chapter,  we  first  review  classical  framework  and  modeling  of  flow  control  problem, 
and  discuss  the  problem  of  TCP/TFRC  over  wireless,  from  a  theoretical  point  of  view. 
Then  we  propose  a  new  solution,  demonstrate  its  connections  to  the  existing  framework, 
and  show  that  it  meets  all  of  our  desired  flow  control  goals.  Some  practical  concerns, 
such  as  the  effects  of  delay  and  incremental  deployment  of  the  scheme,  are  also  addressed 
by  theoretical  analysis.  Finally,  we  show  the  solution  is  an  application  of  a  general  two 
timescale  framework  for  flow  control.  In  this  framework,  network  and  users’  dynamics 
evolve  in  two  different  time  scales,  and  control  laws  are  designed  for  the  dynamics  in  each 
timescale.  Sufficient  conditions  for  the  dynamics  to  converge  to  a  desired  equilibrium  are 
characterized.  One  advantage  of  this  framework  is  that  it  potentially  allows  us  to  solve 
any  flow  control  problem  without  modifying  underlying  network  infrastructure  or  transport 
layer  protocol  stack. 
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3.1  Overview  of  The  Flow  Control  Framework  and  TCP 


Modeling  over  Wireless 

Consider  a  network  with  a  set  J  of  resources,  i.e.  links,  and  let  Cj  be  the  finite  capacity 
of  resource  j,  for  j  £  J.  Let  R  be  the  set  of  routes,  where  a  route  r  is  a  non-empty  subset 
of  J  associated  with  a  positive  round  trip  delay  Tr.  We  assume  Tr  is  fixed;  this  is  a  quite 
natural  and  common  assumption,  at  least  for  the  time  scales  we  are  interested  in,  as  argued 
in  [29]  and  [26].  Set  cijr  =  1  if  j  £  r,  and  set  ajr  =  0  otherwise.  This  defines  a  0-1  routing 
matrix  A  =  ( a,jr,j  £  J,  r  £  R),  indicating  the  connectivity  of  the  network. 

Associate  a  route  r  with  a  user,  i.e.  a  pair  of  sender  and  receiver,  and  assume  users 
behave  independently;  furthermore,  endow  a  user  with  a  sending  rate  xr  >  0  and  a  util¬ 
ity  function  Ur(xr),  which  is  assumed  to  be  increasing,  strictly  concave,  and  continuously 
differentiable.  For  convenience,  we  also  define  x  to  be  a  vector  of  users’  sending  rates,  i.e. 
x  =  [xi,x2,  ■  •  -]T. 

Assume  utilities  are  additive,  so  that  the  aggregate  utility  of  the  entire  system  is 
^2reR^r(xr).  The  flow  control  problem  under  a  deterministic  fluid  model,  first  introduced 
by  Kelly  et.  al.  [29]  and  later  refined  in  [28],  is  a  concave  optimization  problem  maximizing 
the  net  utility1: 

— -V  r 

Ur(xr)  —  /  pj(z)dz,  (3. 

r&R  j£j 

xs 

where  f0  s:je®  '  Pj(z)  dz  can  be  considered  as  the  cost  incurred  at  link  j;  Pj(z)  is  called  the 
price  function  and  is  required  to  be  non-negative,  continuous,  increasing,  and  not  identically 
zero.  With  these  assumptions  on  Pj(z),  the  objective  function  in  Equation  (3.1),  the  sum 
of  users’  utilities  minus  the  costs  associated  with  using  the  links,  is  strictly  concave.  One 
common  price  function  used  in  practice  is  the  packet  loss  rate,  which  is  zero  if  there  is  no 

1It  is  easy  to  show  that  this  is  nothing  but  a  penalty  relaxation  solving  the  sum  utility  maximization 
with  capacity  constrains,  namely: 

maxj>0  J^UriXr) 
reR 

s.t.  Ax  <  C,  (i.e.  Y.s:jesx‘>  ^  ci,3  €  J ) 
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congestion,  and  concavely  increases  otherwise: 


Pj(z) 


(*  ~  C,)  + 

z 


0, 


z  <Cj\ 


c 


1-^,  z>C,j. 


(3.2) 


where  Cj  is  the  capacity  of  link  j .  For  ease  of  use  in  the  remainder  of  the  paper,  define  the 
aggregate  rate  arriving  at  link  j  as  follows: 


Vj(t)  =  xs(t),  j  G  J. 

s:jGs 

In  the  rest  of  this  thesis,  we  assume  Pj{yj{t))  is  small  enough  such  that  the  end-to-end 
packet  loss  rate  for  user  r,  i.e.  1  —  fl (1  ~  Pj{yj{t))),  is  approximately  ^2j&rPj(yj(t)). 


Under  these  settings,  Kelly  [28]  has  shown  TCP  Reno  2  to  be  a  primal-like  algorithm, 
with  xr(t)  satisfying: 


®r(*)  =  9^  (  ~  xr^  |  ’  r  G  R> 

r  j&r 


(3.3) 


and  solving  the  optimization  problem  in  Equation  (3.1)  with  Ur(xr )  =  —  where  S  is  the 
TCP  packet  size,  Tr  is  the  end-to-end  round  trip  time,  and  Pj(y)  takes  the  form  in  Equation 
(3.2).  Rewriting  Equation  (3.1)  with  these  quantities,  the  net  utility  function  becomes: 


max  <  — 


reRIrXr  jeJJo  Z 


(3.4) 


Thus,  TCP  can  be  viewed  as  a  discrete  version  of  the  steepest  gradient  descent  algorithm 
which  solves  the  optimization  problem  in  Equation  (3.4). 


From  control  perspective,  a  network  with  TCP  users  can  be  modeled  as  a  feedback 
control  system,  as  shown  in  Figure  3.1.  TCP  user  r  uses  its  sending  rate  xr(t)  as  an  input 
into  the  network,  and  adjusts  xr(t)  dynamically  according  to  the  feedback,  i.e.  congestion 
loss  in  Figure  3.1. 


Equation  (3.3)  is  a  differential  equation  describing  the  time  evolution  of  sending  rate 
xr(t),  whereby  the  user  exploits  only  the  aggregate  packet  loss  information  along  its  path. 
While  routers  drop  packets  and  feedback  the  packet  loss  rate  to  TCP  users  based  on  only  the 
2In  the  rest  of  the  thesis,  we  use  this  version  of  TCP. 
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Figure  3.1.  Feedback  control  modeling:  TCP  over  wired  networks 

difference  between  aggregate  incoming  rates  and  outgoing  links  capacities,  both  users  and 
routers  operate  based  on  only  local  information;  hence,  distributed  algorithms  are  possible 
to  carry  out  the  operations.  Specifically,  users  use  TCP,  and  routers  use  DropTail  or  RED. 

In  the  analysis,  the  same  flow  xr(t)  is  assumed  to  be  presented  to  all  links  j  6  r,  even 
though  the  flow  in  downstream  links  slightly  shrinks  due  to  loss  at  upstream  links.  This 
is  a  direct  implication  of  our  previous  assumption  that  the  packet  loss  rate  on  link  j,  i.e. 
Pj(52s:jesX*)’  is  smalL 

Kelly  [28]  showed  the  system  in  Equation  (3.3)  has  a  unique  equilibrium,  to  which  all 
trajectories  converge,  as  follows: 

x°  =  -  ,  r  6  R;  (3.5) 

Tr^jerPj  (yj) 

where  y°  =  Yls-j&s  x° ■  Equation  (3.5)  is  similar  to  the  well  known  TCP  steady  state 
throughput  equation  as  described  by  Mahdavi  and  Floyd  in  [37].  TCP  in  wired  networks 
achieves  all  desired  flow  control  goals: 

•  Algorithms  of  users  and  routers  are  distributed. 

•  Users’  sending  rate,  controlled  by  TCP,  converge  to  a  unique  equilibrium  exponentially 
fast  [32], 


24 


•  Upon  the  unique  equilibrium,  the  optimization  problem  in  Equation  (3.1)  is  solved, 
and  net  utility  is  maximized. 

•  Upon  the  unique  equilibrium,  all  bottleneck  links  are  fully  utilized.  This  can  be  seen 
from  Equation  3.5,  where  finite  value  of  x°  implies  positivity  of  end-to-end  packet  loss 
rate  Ylj&Pj  (y°)>  which  in  terms  indicates  the  bottleneck  of  route  r  must  be  fully 
utilized  and  congested. 

•  Upon  the  unique  equilibrium,  there  is  roughly  a- fairness  [38]  among  users  with  a  =  2, 
implying  the  users  are  allocated  rates  that  roughly  maximize  a  sum  of  utility  of  the 
form  xj~a  =  x~x . 

3.2  TCP  and  TFRC  over  Wireless 

For  wireless  networks,  we  assume  the  links  j  £  •/  are  associated  with  not  only  a  fixed 
capacity  but  also  a  packet  loss  rate  caused  by  the  physical  channel  errors,  necessarily  non¬ 
negative,  denoted  by  tj  >  O.j  e  J .  Nevertheless,  TCP  over  wireless  is  still  associated  with 
the  same  concave  optimization  problem  as  with  the  wired  networks,  shown  in  Equation 
(3.4).  This  is  because  neither  the  utility  function  of  users  in  Equation  (3.4),  i.e.  Ur(xr), 
nor  the  cost  associated  with  using  network  resources,  i.e.  Pj(z)  dz,  is  a  function  of 
£j  >  0 ,j  6  J.  In  effect,  €j  results  in  the  price  functions  fed  back  to  users  to  be  inaccurate, 
since  it  now  includes  loss  both  due  to  congestion  and  physical  channel  errors.  Hence  TCP 
algorithms  still  aim  to  address  the  same  optimization  problem  shown  in  Equation  (3.4),  but 
with  inaccurate  prices  fed  back  from  network. 

From  control  perspective,  as  shown  in  Figure  3.2,  presented  wireless  channel  error  makes 
the  feedback  (price)  from  network  to  users  inaccurate.  Users  control  sending  rates  based 
on  the  inaccurate  feedback;  as  a  result,  the  controlled  sending  rates  might  converge  to  an 
equilibrium  that  causes  underutilization  in  links. 

This  inaccurate  feedback  price  function,  denoted  by  qj(yj(t)),  is  the  sum  of  ej  and 
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Figure  3.2.  Feedback  control  modeling:  TCP  over  wireless  networks 


Pj(jjj(t)),  under  the  assumption  that  ej  is  small 

QjiVjit))  =  pj  ( yj(t ))  +  ej  >  €j,  j  <5  J. 


(3.6) 


When  the  link  is  not  congested,  qj(yj(t))  =  ej  since  all  packet  loss  are  caused  by  channel 
error;  qj(yj(t ))  gradually  increases  otherwise.  With  this  inaccurate  price,  TCP  now  adjusts 
the  sending  rates  as: 


.  .  ,  1  /  25*2 

xr(t)  =  - 


T? 


x2r{t)  qj(yj(t ))  I  ,  r  <E  R.  (3.7) 

j&r  J 

Following  a  similar  analysis  in  [28],  one  can  show  the  system  in  Equations  (3.7)  and  (3.6) 
has  a  new  unique  equilibrium,  to  which  all  trajectories  converge,  as  follows: 


xr  = 


Vis 


< 


Vis 


reR, 


(3.8) 


Tr  y  Hj£r  Qj  (Vj  )  Tr  y  ej 

where  fjj  =  Although  it  can  be  shown  that  there  is  roughly  a-fairness  among 

users  with  a  =  2,  x  =  [xr,r  G  R]  is  a  suboptimal  solution  as  it  is  different  from  the  unique 
optimal  one  x°.  Furthermore,  user  r  could  suffer  underutilization  if  Yljer  ej  sufficiently 
large.  For  instance,  in  the  one  user  one  bottleneck  network,  underutilization  happens  if 
and  only  if  VVSr  <  where  C  is  the  bottleneck  bandwidth  and  e  represents  the  aggregate 

J-ry/6 


packet  loss  rate  caused  by  wireless  channel  error  experienced  by  the  user,  as  shown  in 
previous  chapter.  Hence  the  main  problem  with  TCP  over  wireless  is  underutilization  of 
the  wireless  channel;  in  fact,  similar  analysis  shows  that  it  is  also  the  main  problem  with 
any  flow  control  method  that  uses  packet  loss  rate  as  price  function,  such  as  TFRC. 
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One  straightforward  solution  is  to  provide  user  r  with  the  accurate  price 
and  apply  it  to  the  control  law  of  xr(i);  this  could  be  done  by  either  end-to-end  estimation 
with  or  without  cross  layer  information,  or  by  hiding  the  wireless  loss  from  users  via  local 
retransmissions.  In  fact,  most  existing  approaches  belong  to  this  class  of  solutions,  thus 
requiring  modifications  either  to  the  transport  protocols  or  to  the  network  infrastructure, 
making  them  hard  to  deploy  in  practice. 

In  essence,  the  main  challenge  of  TCP  optimization  problem  shown  in  Equation  (3.4) 
for  the  wireless  network  setting  is  to  accurately  estimate  price  Pj(yj(t))  from  the  noisy 
measurements  Qj(yj{t)).  One  way  to  overcome  this  problem  is  to  specifically  choose 
a  slightly  different  utility  maximization  problem,  resulting  in  a  new  solution  that  requires 
measurements  that  are  easy  to  obtain  in  a  practical  networking  setup.  In  the  next  section, 
we  will  show  that  by  selecting  a  different  utility  to  maximize,  there  exists  a  new,  stable  and 
optimal  solution,  which  requires  only  one  bit  of  end-to-end  measurement. 


3.3  Proposed  solution 

3.3.1  A  New  Class  of  Solutions 

Motivated  by  insights  provided  by  previous  discussion  in  Section  2.2,  we  now  propose 
a  new  approach  to  flow  control  based  on  adjusting  the  number  of  connections  for  user  r, 
denoted  by  nr{t).  This  is  an  end-to-end  application  layer  based  scheme,  and  requires  no 
modification  to  the  network  infrastructure  or  the  transport  protocol  stack. 

In  Section  2.2,  multiple  TFRC  connections  are  used  to  transmit  one  video  stream. 
Sending  rate  of  individual  connections  is  controlled  by  TFRC  itself.  We  have  argued  that 
the  number  of  connections  nr(t )  should  be  controlled  to  pursue  an  optimal  value,  and  one 
strategy  would  be  to  increase  nr(t )  until  congestion  is  observed.  The  NS-2  simulations  and 
actual  experiments  in  Section  2.2  show  the  performance  of  such  an  approach.  Motivated  by 
this  insight,  our  proposed  approach  is  to  dynamically  adjust  both  xr(t)  and  nr(t). 
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In  our  approach,  user  r’s  rate  xr(t)  is  the  aggregate  rate  of  nr(t)  TCP  connections 
user  r  opens.  The  dynamics  of  xr(t)  is  similar  to  that  of  an  individual  TCP,  except  it  is 
more  aggressive  upon  no  packet  loss  and  less  conservative  upon  packet  loss.  Specifically, 
when  there  is  no  packet  loss,  xr(t)  increases  by  nr(t)S/Tr  per  round  trip  time  Tr  rather 
than  by  S/Tr  per  Tr  as  is  the  case  with  a  single  TCP.  When  a  packet  loss  is  observed, 
xr(t)  decreases  by  a  factor  of  1/2 nr(t)  rather  than  by  1/2  as  in  a  single  TCP  connection 
case.  Note  the  number  of  lost  packets  user  r  observes  during  Tr  is  ^ xr(t)Tr 
when  packet  loss  rate  X^/er  QjiVjit))  is  small;  hence,  xr(t)  changes  approximately  — 

2^}ggxr(t)rr  Yljer  in  a  round  trip  time  Tr.  As  a  result,  the  control  law  of  xr(t) 

can  be  expressed  as  follows: 


Xr(t)=  (  2S  ^ )  -  x2r  (t)  J2  Qj  (Vj  (*) )  1 


?2nr2 

’J '2 


(3.9) 


jer 


The  basic  idea  of  controlling  nr(t )  is  to  increase  it  if  there  is  no  congestion,  and  de¬ 
crease  it  otherwise.  In  our  approach,  we  apply  Inverse  Increase  and  Multiplicative  Decrease 
(HMD)  for  nr(t),  i.e.  nr(t)  is  increased  inversely  when  there  is  no  congestion,  and  decreases 

multiplicatively  otherwise.  The  explicit  control  law  is  shown  below: 

/  \ 

1 


nr(t) 


nr{t)Ir[^2pj{yj(t ))]  ,  r  e  R 


(3.10) 


j£r 


where  cr,r  6  R  are  nonnegative  constants  indicating  how  fast  nr(t),r  G  R  are  adjusted, 
and  Ir(Zj&Pj(Vjm  an  indicator  function  implying  the  congestion  status  of  route  r: 

~  Cj)+ 


j&r 


j'6r 


yj(t) 


(3.11) 


=  < 


1,  if  route  r  is  congested  at  time  t, 

i  p  V  >  n. 

Z-'j&r  yj(t)  >  U’ 


0,  otherwise. 

As  we  will  see  later,  this  law  leads  the  system  to  a  stable  equilibrium  that  meets  all  of  our 
design  goals. 

As  shown  in  Equation  (3.9)  and  explained  above,  the  control  law  of  the  aggregate  rate  for 
user  r,  i.e.  xr(t),  can  be  understood  as  the  sum  of  rates  from  nr{t )  individual  connections, 
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with  each  connection  being  controlled  using  the  standard  TCP  Reno  algorithm.  Therefore, 
our  approach  does  not  require  any  modifications  to  the  TCP  protocol.  On  the  other  hand, 
as  seen  in  Equation  (3.10),  the  system  tries  to  achieve  full  utilization  by  adjusting  the 
number  of  connections  nr(t)  accordingly.  In  particular, 

•  If  a  route  r  is  underutilized,  then  Ir[%2j£rPj(yj(t))]  =  0;  this  implies  that  the  number 
of  connections  nr(t)  will  increase  in  order  to  boost  the  user’s  rate  xr(t),  pursuing  full 
utilization  on  any  route  r; 

•  If  the  route  r  is  fully  utilized,  i.e.  one  of  its  links  is  congested,  then  4- Ejer  Pj(yj(t))\  = 
1,  lowering  nr(t),  and  hence  xr(t),  to  prevent  the  system  from  further  congestion. 

The  intuition  behind  our  approach  is  as  follows:  when  loss  rate  caused  by  channel  error 
increases,  individual  connection’s  sending  rate  is  lowered,  thus  users  need  to  open  more 
connections  to  increase  the  aggregate  throughput.  The  Ir(')  is  the  one  bit  of  information 
required  from  the  end-to-end  measurements.  In  practice,  lots  of  techniques  can  be  applied 
to  estimate  Ir(-)  using  end-to-end  statistics  [11,  12,  14,  20,  33,  34,  42,  45,  47,  48,  51,  57].  In 
particular,  we  estimate  the  queuing  delay  by  comparing  current  round  trip  time  with  the 
propagation  delay,  and  set  Ir(-)  =  1  if  the  queuing  delay  is  positive,  and  Jr(-)  =  0  otherwise. 

The  system  in  Equations  (3.9)  and  (3.10)  is  a  coupled,  discontinuous,  nonlinear  system. 
In  order  to  verify  that  this  system  actually  meets  our  design  goals,  we  need  to  analyze  the 
existence  of  a  unique  equilibria,  its  stability  and  its  optimality  in  the  sense  of  solving  a 
utility  maximization  problem. 

3.3.2  Discontinuity  Approximation  and  The  Two  Timescale  Decomposi¬ 
tion 

The  discontinuities  of  functions  PrEjer2h'(26'(^))]  anc^  Pjiyjffl)  hinder  the  analysis  of 
the  equilibria.  To  carry  out  the  analysis,  we  first  approximate  these  discontinuous  functions 
using  continuous  functions,  in  order  to  generate  an  approximate  continuous  system  to  the 
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original  discontinuous  system3:  Vj  £ 


1 


Pj(Vj(t))  ~  ^ln  (  1  +  e  Vj 


ft  Vj  W—Cj 

P  y,(t)  I  A 


=  9 


(3-12) 


_  eP'Ejer9j{yj(t))  _  1 

53  ^Eje,a(a(.))  +  1  -  f(Esi(y,m 


(3.13) 


jer  jer 

where  (3  is  a  nonnegative  constant.  It  should  be  clear  that  f(^2j&r  9j(Uj(t)) 


Ir{J2j€r9j{yj(t))  and  gj{yj{t ))  -»•  Pj(yj(t))  as  f3  -►  oo. 


Thus,  an  approximate  continuous  version  of  the  original  system  in  Equations  (3.9)  and 
(3.10)  is  given  by:  Vr  E  i?, 

M*)  =  2sht}  -  Xr^  Ejerfo  +9j(yj(t))]\  , 

<  /  ,  \  (3-14) 

nr{t)  =  cr  I  ^  -  nr(t)f(£lj&9j(yj(t)))  \  - 

Since  the  approximate  system  in  Equation  (3.14)  is  continuous,  we  can  analyze  its  equi¬ 
librium  and  stability  for  arbitrary  values  of  (3.  As  (3  — ►  oo,  the  system  in  Equation  (3.14) 
approaches  the  original  system  in  Equations  (3.9)  and  (3.10).  Therefore,  the  equilibria 
and  the  stability  results  for  the  approximate  system  also  correspond  to  the  results  for  the 
original  system  in  the  Filippov  sense  [30,  46]. 


The  approximate  system  in  Equation  (3.14),  though  continuous,  is  difficult  to  analyze 
in  general.  It  is  a  nonlinear,  coupled,  multivariate  system,  and  the  two  equations  are  not 
exactly  symmetric  even  though  they  might  appear  to  be  so.  Hence,  we  introduce  a  two  time 
scale  assumption  to  analyze  the  approximate  system:  The  number  of  connections,  nr(t), 
changes  in  a  timescale  much  slower  than  the  source  rate,  xr(t).  This  assumption  is  the 
key  not  only  for  carrying  out  the  equilibria  and  stability  analysis,  but  also  for  extending 
the  results  and  opening  the  door  to  a  general  two  time  scale  framework  for  flow  control, 
as  we  will  see  later.  It  is  also  reasonable  in  practice,  where  the  sending  rates  are  expected 
to  change  on  the  order  of  tens  of  milliseconds,  while  number  of  connections  is  expected  to 
change  at  a  much  slower  rate,  e.g.  tens  of  seconds. 


Under  the  above  assumption,  system  in  Equation  (3.14)  fits  into  the  classical  singular 

perturbation  framework  [46]  [30],  and  therefore  can  be  decoupled  into  a  fast  timescale  and 

3We  will  discuss  the  relationship  between  the  approximate  system  and  the  original  system,  and  perfor¬ 
mance  of  the  actual  implementation  later. 
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a  slow  timescale  system.  The  fast  timescale  system  is  described  by  Equation  (3.9)  with  the 
corresponding  nr(t),  r  £  R  being  constant,  namely  boundary  system:  Vr  £  R, 

*»■(*)  =  2 sht)  {^TT1  ~~  *?(*)  Eierfo  +  *■(%■(<))])  ,  (3  15) 

nr(i)  =  constant. 

The  slow  timescale  system  is  described  by  Equation  (3.10)  along  the  equilibrium  manifold 
defined  by  the  stationary  solution  of  Equation  (3.9),  namely  reduced  system:  Vr  £  R, 


xr(t)  = 


nr(t)y/2S 

Try/'Ejer[tj+9j(yj(t))]  ’ 


M 


(*)  =  Cr  (  “  nr(i)/(Eier  0J  (%'(*)))  )  ■ 


(3.16) 


jSr 


Under  the  two  timescale  setting,  the  behavior  of  the  system  can  be  described  as  follows. 
On  the  fast  timescale,  n(t)  can  be  thought  of  as  being  constant  since  its  dynamics  happens 
at  a  slow  timescale.  The  entire  system  can  then  be  expressed  as  a  boundary  system  shown 
in  Equation  (3.15),  and  only  dynamics  of  x(t)  is  considered.  As  the  boundary  system  is 
similar  to  Kelly  et.  al.’s  control  system  on  wired  networks  [28]  except  for  the  constant  n(t), 
and  the  price  function  Pj(yj(t ))  being  replaced  by  tj  +  gj(y(t),  the  behavior  of  the  boundary 
system  can  be  easily  inferred  from  the  known  results  for  the  system  in  Equation  (3.3). 
Specifically,  at  the  fast  timescale,  x(t)  =  [xr(t),r  £  R]  globally  exponentially  converges  to 
the  equilibrium  manifold  defined  as  follows  [28,  32]: 

xr(t)  =  ,  nAt)^S  ter.  (3.17) 

idj  (Vj(t))  +  ej\ 

This  can  be  easily  shown  be  to  a  one-to-one  mapping  between  x(t)  and  n(t)  =  [nr(t),r  £  R]: 


Lemma  3.3.1.  The  equilibrium  manifold  defined  by  Equation  (3.17),  is  a  one-to-one  map¬ 
ping  between  x(t)  and  n(t). 


Proof.  On  one  hand,  it  is  easy  to  see  directly  from  Equation  (3.17)  that  one  set  of  x(t) 
results  in  one  set  of  n(t).  On  the  other  hand,  given  a  set  of  n(t),  Equation  (3.17)  is  the 
solution  solving  the  following  strictly  concave  optimization  problem: 

2  n2r(t)S2  ^  rY. 


max  <  — 


E 

r£R 


Tfx 


r 


E 

3£J' 


s:jes  - 


[o  +  9j  ( Vj(t ))]  dz  V  , 


(3.18) 
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and  the  solution  must  be  unique.  Therefore,  one  set  of  n(t)  only  results  in  one  set  of  x(t), 
and  the  mapping  is  one-to-one.  □ 

On  the  slow  timescale,  x(t)  has  already  converged  to  the  above  equilibrium  manifold, 
and  now  the  system  collapses  into  the  reduced  system  described  in  Equation  (3.16).  x(t)  has 
already  converged  onto  the  equilibrium  manifold,  and  only  dynamics  of  n(t)  are  explicitly 
considered,  whose  behavior  determines  how  the  approximate  system  evolves  at  the  slow 
timescale.  Therefore,  together  with  boundary  system,  it  fully  characterizes  behavior  of  the 
system  for  all  possible  timescales.  For  illustration  purposes,  we  have  handdrawn  the  time 
dynamics  of  singular  perturbation  system  comprising  of  a  single  user  with  sending  rate  x±(t), 
and  n\[t)  connections  in  Figure  3.3.  As  shown,  given  any  initial  condition,  the  system  first 
converges  rapidly  onto  the  equilibrium  manifold,  representing  the  fast  timescale  indicated 
by  its  boundary  system.  On  the  manifold,  the  system’s  behavior  follows  its  reduced  system; 
if  the  reduced  system  has  the  equilibrium  as  the  globally  asymptotically  stable  equilibrium, 
then  the  trajectories,  along  the  manifold,  will  converge  to  the  equilibrium  as  time  goes  to 
infinity. 

In  the  next  subsection,  we  present  the  results  on  the  existence  of  a  unique  optimal 
equilibrium  of  the  system  and  its  stability. 

3.3.3  The  Existence  of  A  Unique  Optimal  Equilibrium  and  Its  Stability 

Given  the  system  in  Equation  (3.14),  the  first  question  to  answer  is  whether  it  has  any 
equilibria,  and  if  so  how  many.  The  second  question  is  the  local  and  global  stability  of  these 
equilibria.  These  two  questions  are  important  in  the  sense  that  they  describe  the  behavior  of 
the  system  as  time  evolves,  predicting  the  system’s  performance  in  actual  implementations 
in  practice.  For  instance,  if  the  system  in  Equation  (3.14)  has  no  equilibrium,  the  users’ 
sending  rates  would  not  converge,  making  the  system  undesirable  to  implement. 

First,  we  define  four  notions  of  stability  to  be  used  in  the  following  theorems  and  lemmas, 
as  follows: 
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Figure  3.3.  Time  dynamics  of  a  singular  perturbation  system. 

Definition  3.3.1.  Local  Stability:  an  equilibrium  £*  of  a  continuous  system  f  =  </?(£,  o) 
is  locally  stable,  if  such  that  all  system  trajectories  starting  from  any  £o?  satisfying 
|£o  —  £*|  <  b,  converge  uniformly  and  exponentially  to  £*. 

Definition  3.3.2.  Global  Exponential  Stability:  an  equilibrium  of  a  continuous 
system  f  =  tp(£,t,£ o)  is  globally  exponentially  stable,  if  all  system  trajectories  starting  from 
any  fo,  converge  uniformly  and  exponentially  to  f*. 

Definition  3.3.3.  Semi  Global  Exponential  Stability:  an  equilibrium  f*  of  a  contin¬ 
uous  system  f  =  ip(f,  t ,  £o)  is  semi  globally  exponentially  stable,  if  all  system  trajectories, 
starting  from  any  fo,  converge  exponentially  but  not  uniformly  to  £*.  That  is,  the  conver¬ 
gence  rate  depends  on  the  initial  condition  £o- 

Definition  3.3.4.  Global  Asymptotical  Stability:  an  equilibrium  £*  of  a  continuous 
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system  f  =  <p(£,  t,£o)  is  globally  asymptotically  stable,  if  all  system  trajectories,  starting 
from  any  fo,  asymptotically  converge  to  £*. 

We  now  show  that  the  system  in  Equation  (3.14)  has  only  one  unique  equilibrium,  which 
is  locally  exponentially  stable,  and  semi  globally  exponentially  stable,  i.e.  it  is  exponentially 
stable  as  long  as  n(t)  and  x(t)  are  constrained  to  lie  in  a  compact  set4.  Further,  the  unique 
equilibrium  solves  a  concave  optimization  problem.  We  show  this  through  the  following 
theorem. 


Theorem  3.3.1.  For  arbitrary  f3  >  0,  the  approximate  system  in  Equation  (3. If)  has  a 
unique  equilibrium,  denoted  by  ( x*,n *),  given  by 


71  =  - _ 

'■  v/7(E f^fwy 


r  €  R: 


V2S 


(3.19) 


=  ,  r  €  R. 


y/f(.T,jer  9j  (Vj  ))Tr  [dj  (Vj  )  +£i] 

Further,  this  unique  equilibrium  solves  the  following  concave  optimization  problem 


Vj 


max 

x>0 


E  Ur(Xr)  ~  E  /  9j{z)  dz, 


reR 


J£J' 


with  Ur  being  concave  function: 


rXr  (  2  S2 

Ur(xr)  =  I  K1  )  dv,  r  G  R, 


(3.20) 


where  hr  1  is  the  inverse  of  a  monotonically  increasing  function  hr: 


Ma>  =  (  E +  *)/(*)  =  (  Eei  +  2)^feWY’  reR- 

j&r  '  ^  j&r  7 


ePz  _  1 


Proof.  Refer  to  Appendix  A. 

4Note  n{t)  and  x(t)  are  typically  constrained  in  practice. 


□ 
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Two  observations  to  be  made  from  Theorem  3.3.1  are  the  following.  First,  for  large  (3, 
bottleneck  links  for  every  user  are  almost  fully  utilized  at  the  equilibrium  shown  in  Equation 
(3.19).  This  is  because  at  the  equilibrium,  h*  =  0 ,r  £  R ;  hence,  fiYljerdjiVj))  =  l/(n*)2  > 
0,  r  £  R,  from  Equation  (3.14).  This  implies  that  at  the  equilibrium  f(Yljer  9j(Vj)) 
strictly  positive.  Taking  into  account  Equation  (3.13)  for  large  f3 ,  /(Xyer  >  0 

indicates  Yljer 9j{v*j)  >  0)  be.  the  packet  loss  rate  caused  by  congestion  is  also  strictly 
positive.  This  implies  that  at  least  at  one  link  j  along  user  r’s  path,  the  aggregate  rate  y* 
passing  through  link  j  is  close  to  or  lager  than  the  link  capacity  Cj ,  hence  bottleneck  being 
almost  fully  utilized. 

Second,  the  unique  equilibrium  for  the  system  in  Equation  (3.14)  in  wireless  scenario 
solves  a  concave  optimization  problem  similar  to  the  one  TCP  Reno  solves  in  the  wired 
network.  They  have  the  same  form  as  Equation  (3.1)  but  with  different  users’  utility 
functions  Ur(xr).  The  only  difference  is  that  the  Ur{xr )  in  the  wired  case  is  only  a  function 
of  xr,  while  in  wireless  scenario  it  is  also  a  function  of  be.  the  Packet  loss  rate 

associated  with  the  route  r.  In  fact,  if  we  let  /3  — >  oo  and  ej  =  0,  Vj  £  J,  i.e.  in  the  wired 
network  scenario,  we  have  hr(z )  =  z.  and  the  optimization  problem  in  Equation  (3.20) 
becomes  identical  to  the  TCP  optimization  problem  in  Equation  (3.4).  In  this  case,  the 
equilibrium  (x*,n*)  is  exactly  the  same  as  x°,  implying  TCP  optimization  problem  in  the 
wired  network  is  merely  a  special  case  of  that  in  Equation  (3.20). 

Given  the  system  in  Equation  (3.14)  has  a  unique  optimal  equilibrium,  an  important 
question  to  answer  is  that  whether  it  is  stable,  i.e.  will  the  users’  rates  converge  to  it.  The 
following  two  theorems  explore  the  answer  to  this  question. 

Theorem  3.3.2.  For  arbitrary  f)  >  0,  under  the  two  timescale  assumption,  the  unique 
equilibrium  of  the  reduced  system  in  Equation  (3.16)  is  locally  exponentially  stable.  Also, 
the  unique  equilibrium  of  the  approximate  system  in  Equation  (3. If),  (x*,n*),  is  locally 
exponentially  stable. 
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Proof.  Refer  to  Appendix  B. 


□ 


Theorem  3.3.2  implies  that  if  the  number  of  connections  n(t)  is  initially  in  a  small  ball 
around  the  equilibrium  n*,  then  the  entire  system  will  converge  exponentially  fast  to  the 
equilibrium.  As  no  convergence  can  be  faster  than  exponential,  this  is  the  best  result  one 
can  expect  in  a  local  region  around  the  equilibrium. 

How  about  when  n(t)  starts  far  away  from  the  equilibrium  n*l  To  answer  this  question, 
we  explore  the  global  stability  of  the  equilibrium. 

We  state  three  lemmas  that  are  needed  for  proving  the  global  exponential  stability.  The 
first  lemma  explores  an  interesting  structure  of  the  vector  field  of  the  boundary  layer  system 
and  the  reduced  system. 

Lemma  3.3.2.  There  exists  a  compact  set,  denoted  by  fR,  for  n(t)  in  the  reduced  system  in 
Equation  (3.16)  with  arbitrary  [3  >  0,  such  that  any  compact  set  containing  it  is  a  positively 
invariant  one.  The  same  observation  is  also  true  for  x(t)  in  the  boundary  layer  system 
Equation  (3.15),  and  the  corresponding  compact  set  is  defined  as  rR(n),  a  function  of  n. 


Proof.  Refer  to  Appendix  C.  □ 

A  positive  invariant  set  is  a  set  with  all  trajectories  on  its  boundary  pointing  inwards; 
as  such,  no  trajectories  inside  the  set  will  ever  move  out. 

The  next  lemma  investigates  the  global  asymptotical  stability  of  the  equilibrium  in 
the  reduced  system.  The  particular  non-linear  shape  of  the  vector  field  for  hr(t)  shown 
in  Equation  (3.16)  makes  the  search  for  a  suitable  Lyapunov  function,  or  a  function  on 
which  to  apply  the  La  Salle  principle,  a  challenging  task.  We  have  found  that  none  of  the 
techniques  applied  in  [29]  and  [17]  work  in  this  case.  We  therefore  believe  that  our  functions 
for  applying  the  La  Salle  principle,  in  the  proof  of  following  lemma,  may  provide  new  insight 
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to  searching  Lyapunov  functions,  or  functions  for  the  La  Salle  principle,  in  similar  or  general 
cases. 

Lemma  3.3.3.  The  unique  equilibrium  of  reduced  layer  system  in  Equation  (3.16),  with 
arbitrary  f3  >  0,  is  a  globally  asymptotically  stable  one. 


Proof.  Refer  to  Appendix  D.  □ 

The  third  lemma  states  that  for  continuous  systems,  local  exponential  stability  and 
global  asymptotical  stability  is  equivalent  to  semi-global  exponential  stability.  We  should 
not  here  that  this  lemma  is  quite  general  and  as  such,  its  use  is  not  restricted  to  the  use  in 
particular  problem  discussed  in  this  thesis. 

Lemma  3.3.4.  Consider  a  system  f  =  ip(£,  t,  £o)  satisfying  the  following  assumptions: 

•  it  has  a  unique  equilibrium  at  0  that  is  locally  exponentially  stable  and  globally  asymp¬ 
totically  stable; 

•  </?(£>  t,  £o)  is  continuous. 

Then  the  equilibrium  of  the  system  is  semi-globally  exponentially  stable. 


Proof.  Refer  to  Appendix  E.  □ 

In  Lemma  3.3.4,  semi-global  exponential  stability  of  an  equilibrium  of  the  system  implies 
starting  from  arbitrary  initial  point,  the  trajectory  of  the  system  will  always  converge  to 
the  equilibrium  exponentially  fast.  However  the  convergence  rate  depends  on  the  initial 
point,  i.e.  the  convergence  is  not  uniform.  Clearly,  semi-global  exponential  stability  is  a 
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much  stronger  than  local  exponential  stability,  but  it  less  strong  than  global  exponential 
stability  in  the  sense  that  the  semi-globally  exponential  convergence  is  not  a  uniform  one. 

These  three  lemmas  enable  us  to  assert  the  semi-globally  exponential  stability  of  the 
equilibrium  of  the  system  in  Equation  (3.14),  as  follows: 

Theorem  3.3.3.  The  unique  equilibrium  of  singularly  perturbed  system  in  Equation  (3.14) 
with  arbitrary  /3  >  0  is  semi-globally  exponentially  stable. 


Proof.  Refer  to  Appendix  F.  □ 

From  practical  point  of  view,  the  above  theorem  implies  that  for  any  number  of  users  in 
a  network  with  any  initial  sending  rates,  users’  rates  will  converge  to  a  unique  equilibrium 
exponentially  fast. 

The  intuition  behind  both  the  local  and  senri-global  convergence  of  the  entire  system 
is  as  follows:  xr(t)  first  converges  in  the  fast  timescale  to  the  equilibrium  of  the  boundary 
system  in  Equation  (3.15),  defined  by  the  equilibrium  manifold  as  in  Equation  (3.17);  then 
in  a  slow  timescale  ,  nr{t)  and  xr{t)  follow  the  control  laws  of  the  reduced  system  in 
Equation  (3.16)  to  converge  to  the  optimal  equilibrium  along  the  manifold.  An  important 
consequence  of  this  convergence  argument  is  that,  a  combination  of  control  law  in  Equation 
(3.10)  on  nr{t)  and  any  flow  control  method  on  xr(t )  resulting  in  the  same  equilibrium 
manifold  as  TCP,  will  retain  the  convergence  behavior  shown  in  Theorems  3.3.2  and  3.3.3. 
Therefore,  it  is  possible  to  extend  all  these  results  to  TFRC  since  it  has  been  shown  in  [23] 
that  TFRC  has  the  same  stationary  behavior  as  TCP. 

In  summary,  we  have  shown  in  Theorems  3.3.1,  3.3.2  and  3.3.3  that: 

•  For  arbitrary  topology,  arbitrary  number  of  users,  and  arbitrary  initial  sending  rates, 
the  sending  rates  controlled  by  proposed  solution  with  arbitrary  (5  converge  to  a  unique 
equilibrium  exponentially  fast. 
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•  For  large  f3,  all  bottlenecks  are  fully  utilized  at  the  equilibrium;  users’  rates  are  allo¬ 
cated  according  to  Equation  3.19. 

•  At  the  equilibrium,  a  net  utility  shown  in  Equation  3.20  is  maximized. 

We  also  know  the  proposed  solution  can  be  implemented  based  on  local  information  only, 
and  is  distributed.  Therefore,  we  have  achieved  all  of  our  desired  flow  control  goals. 

Theorems  3.3.1,  3.3.2  and  3.3.3  state  the  existence  of  a  unique  optimal  equilibrium,  and 
ensure  its  stability  for  the  continuous  approximate  system  in  Equation  (3.14),  for  arbitrary 
values  of  (3.  In  the  limit  as  /?  — >  oo,  the  approximate  system  approaches  the  original 
discontinuous  system  in  Equations  (3.9)  and  (3.10).  Therefore,  for  extremely  large  /3,  we 
expect  the  approximate  system  to  behave  quite  similarly  to  the  original  system,  except  at 
the  discontinuities  yj{t)  =  Cj. 

As  seen  in  the  next  section,  in  actual  implementation  of  the  proposed  system  in  Equa¬ 
tions  (3.9)  and  (3.10),  it  is  necessary  to  discretize  continuous  quantities.  For  instance, 
controlling  nr{t)  is  implemented  by  adjusting  the  number  of  connections,  which  has  to  be 
an  integer  number;  controlling  xr(t )  is  implemented  by  TCP  to  adjust  the  finite  number 
of  packets  to  be  sent  out  in  a  time  interval.  Therefore,  it  is  highly  unlikely  for  the  system 
to  operate  at  discontinuous  points.  From  this  point  of  view,  the  analysis  based  on  the  ap¬ 
proximate  system  is  accurate  enough  to  predict  and  interpret  the  performance  of  the  actual 
implementation  of  the  algorithm  in  practice. 


3.4  Discussions  on  The  Proposed  Solution 

In  the  proposed  solution,  cr  can  be  chosen  in  a  distributed  fashion  in  the  system  shown 
in  Equation  (3.14),  as  long  as  the  two  timescale  assumption  holds.  Practically,  this  implies 
that  each  user  can  adjust  nr{t )  according  to  a  different  rate.  Specifically,  a  global  setting 
among  all  the  users  is  not  necessary.  Furthermore,  allowing  some  of  the  cr  to  be  zero 
represents  a  scenario  according  to  which  the  proposed  scheme  coexists  with  TCP.  In  this 
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situation,  all  theorems  still  hold,  except  for  a  modification  to  Theorem  3.3.1.  More  precisely, 
we  have  the  following  Corollary: 

Corollary  3.4.1.  For  arbitrary  topology,  arbitrary  number  of  users  running  either  TCP  or 
proposed  solution,  and  arbitrary  initial  sending  rates,  the  following  holds: 


Sending  rates  x(t)  converge  to  a  unique  equilibrium: 


Users  running  TCP: 


x„  = 


y/2S 


WE 


jj£r 


9j  [Vi  +€j 


(3.21) 


Users  running  proposed  solution  as  in  Equation  (3.1 4): 

V2  S 


= 


(3.22) 


\J f(Ej&r9j(y*j))Tr^Ejzr  9j  (Vj^j  +  ej 
At  the  equilibrium,  all  bottlenecks  are  fully  utilized  for  large  [3,  and  the  following 
concave  optimization  problem  is  solved: 


Vi 


max 

x>0 


E  Ur(*r)  ~  E,  9j{z)  dz, 


r&R 


jeJ' 


with  Ur  being  concave  function: 


(3.23) 


—  Users  running  TCP: 


Ur(xr)  —  E  T2X  Xr  E  eji 


J.  r 

reR  r  jSr 


Users  running  proposed  solution  as  in  Equation  (3.1  f): 

fXr  (  2  S2 

Ur(xr)  =  /  hfl  f  )  dv, 

Jo 


r  1  T2i/2 


where  h  1  is  the  inverse  of  an  monotonically  increasing  function  hr: 


hr{z)  =  (  E  el  +  z)f(z)  =  (  E  el  + 


j£r 


j&r 


opz  —  1 

4.  1  ' 


r  G  R. 
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An  observation  on  the  fairness  among  users  is  that  at  the  equilibrium,  users  running 
proposed  solution  are  fair  to  users  running  TCP  in  the  following  sense: 

•  In  the  case  that  TCP  operates  in  capacity-limited  region,  i.e.  all  bottleneck  links  are 
fully  utilized  even  if  all  users  running  proposed  solution  only  open  one  connection,  for 
large  values  of  f3,  the  approximated  indicator  function  /(•)  takes  the  value  1.  Users 
running  proposed  solution  will  open  only  one  connection,  and  their  sending  rates  will 
converge  to  the  same  value  as  if  they  were  running  TCP.  In  this  case,  fairness  among 
users  running  TCP  and  proposed  solution  is  the  same  as  fairness  among  TCP  users. 

•  In  the  case  that  TCP  operates  in  channel-error-limited  region,  i.e.  the  bottleneck 
links  of  some  routers  are  underutilized,  if  users  running  proposed  solution  open  one 
connection,  for  large  values  of  (3,  approximated  indication  function  /(•)  takes  on  a 
value  between  0  and  1,  and  approximated  congestion  loss  function  g{  )  takes  on  a  very 
small  positive  value,  as  seen  from  definition  of  /(•)  in  Equation  3.13.  Consequently, 
as  seen  from  users’  equilibrium  rates  in  the  above  Corollary,  users  running  proposed 
solution  will  get  the  residual  bandwidth  without  affecting  TCP  users’s  throughput, 
i.e.  users  running  TCP  get  throughput  close  to  what  they  would  have  gotten  if  no 
other  users  were  running  the  proposed  solution. 

Above  results  imply  that  in  a  network  where  our  schemes  coexist  with  TCP,  all  users’ 
rates  will  again  converge,  semi-globally  exponentially,  to  a  unique  optimal  and  optimistically 
fair  equilibrium.  Note  this  claim  is  true  for  arbitrary  topology,  arbitrary  number  of  users, 
and  arbitrary  initial  sending  rates.  These  observations  on  stability  and  scalability  encourage 
incremental  deployment  of  our  scheme  in  the  current  Internet  where  TCP  is  dominant,  and 
addresses  a  major  concern.  The  simulations  in  Chapter  5  partially  support  this  observation. 

Since  all  analysis  and  results  hold  regardless  of  the  values  of  €j,j  G  J,  it  implies  that 
our  proposed  solution  works  in  both  wired  and  wireless  scenarios.  In  the  wired  scenario, 
because  the  indicator  function  /(•)  takes  value  1,  proposed  solution  ends  up  with  opening 
one  connection  and  reduces  to  traditional  TCP.  Hence,  we  can  afford  to  design  only  one 


41 


Table  3.1.  MATLAB  Simulation  setting. 


Simulation  setting 

Value 

Capacity  of  link  1:  C\ 

100  Bps 

Capacity  of  link  2:  C-2 

160  Bps 

Wireless  packet  loss  rate  on  link  1:  plw 

0.001 

Wireless  packet  loss  rate  on  link  2:  p ^ 

0.0005 

Packet  size:  S 

1000  Bytes 

Round  trip  time  of  user  1:  T± 

2  ms 

Round  trip  time  of  user  2:  T2 

1  ms 

Round  trip  time  of  user  3:  T3 

3  ms 

Adjusting  rate  of  the  number  of  connections  of  user  1:  c± 

0.005 

Adjusting  rate  of  the  number  of  connections  of  user  2:  C2 

0.01 

Adjusting  rate  of  the  number  of  connections  of  user  3:  C3 

0.015 

Simulation  time  interval 

20  ms 

practical  scheme,  following  Equations  (3.9)  and  (3.10),  for  both  wireless  and  wired  networks, 
with  the  latter  corresponding  to  the  case  where  ej  =  0 ,  j  E  J. 


3.5  Illustrated  MATLAB  Simulations 


Figure  3.4.  MATLAB  Simulation  topology. 

To  visually  illustrate  the  system  dynamics  described  in  previous  sections,  we  carry  out 
MATLAB  simulations  for  the  topology  shown  in  Figure  3.4  with  settings  shown  in  Table 
3.1.  As  seen,  ci,C2,C3  are  chosen  to  be  very  small,  in  order  to  satisfy  the  two  timescale 
assumptions. 

In  simulations,  we  control  sending  rates  for  user  i  along  path  S{  to  rj,  i  =  1,2,3,  using 
proposed  scheme  in  Equations  (3.9)  and  (3.10).  The  simulation  results  are  shown  in  Figure 
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so  urce  rates  x 


Figure  3.5.  Illustrated  MATLAB  simulation  results:  sending  rates  and  the  number  of 
connections. 
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3.5,  including  sending  rates  of  users  and  the  number  of  connections.  As  predicted  in  the 
analysis  in  Section  3.3,  proposed  solution  achieves  full  utilization  of  links,  and  users’  sending 
rates  converge  nicely.  Around  full  utilization  of  links,  there  are  some  oscillations  of  aggregate 
rates,  e.g.  aggregate  rate  passing  through  link  2  is  x\ (t)  +  x% (f).  This  oscillation  is  a 
direct  consequence  of  the  gradual  increase  and  sharp  decrease  of  the  number  of  connections, 
according  to  HMD  control  law  shown  in  Equation  (3.10).  Although  only  results  of  one 
instance  of  simulation  are  shown,  the  same  convergence  behavior  of  sending  rates  and  the 
number  of  connections  appears  in  all  other  simulations  with  different  initial  conditions  on 
sending  rates  and  the  number  of  connections. 


3.6  A  General  Two  Timescale  Framework  for  Flow  Control 

In  proposed  solution,  in  order  to  converge  to  a  desired  equilibrium,  two  control  laws 
for  both  number  of  connections  and  sending  rate  of  individual  connection  are  designed. 
Sending  rate  of  individual  connection  is  controlled  by  TCP  to  converge  to  an  equilibrium 
manifold  exponentially  fast,  on  a  fast  timescale.  Number  of  TCP  connections  is  controlled 
to  converge  to  the  desired  equilibrium  exponentially  fast  along  the  manifold,  on  a  slow 
timescale.  We  have  also  argued  that  TFRC  can  be  applied  to  replace  TCP  to  control 
sending  rate  of  individual  connection,  and  the  entire  system  still  converges  to  the  desired 
equilibrium  exponentially  fast. 

This  implies  a  general  two  timescale  framework  as  indicated  from  our  proposed  solution. 
In  this  framework,  it  is  sufficient  to  solve  any  flow  control  problem  by  the  following  process: 

•  Based  on  the  desired  flow  control  goals,  choose  an  equilibrium  of  sending  rates  that 
satisfy  all  specific  goals,  e.g.  full  utilization  of  bottleneck  bandwidth  and  fairness 
among  users. 

•  On  a  fast  timescale,  fix  number  of  connections,  design  a  control  law  for  sending  rate 
of  individual  connections,  such  as  the  one  shown  in  Equation  (3.7),  to  converge  to  a 
equilibrium  manifold  containing  the  desired  equilibrium  exponentially  fast. 
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•  On  a  slow  timescale,  design  a  control  law  for  number  of  connections,  such  as  the  one 
shown  in  Equation  (3.10)  to  converge  to  the  desired  equilibrium  exponentially  fast, 
along  the  equilibrium  manifold. 

If  we  follow  the  above  process,  then  users’  sending  rates  will  converge  to  the  desired  equi¬ 
librium  exponentially  fast.  Theoretically,  this  framework  is  an  application  of  the  singular 
perturbation  theorem  in  [30].  Since  no  convergence  faster  than  exponential  is  possible,  this 
is  in  fact  the  best  convergence  shape  one  can  expect. 

Our  proposed  solution  for  problem  of  flow  control  in  wireless  networks,  follows  this  gen¬ 
eral  framework  implicitly.  By  theorems  and  lemmas  in  this  chapter,  the  proposed  solution 
chooses  an  optimization  problem  to  solve,  of  which  the  optimal  equilibrium  achieves  all 
flow  control  goals.  On  the  fast  timescale,  the  proposed  solution  applies  TCP  to  control 
sending  rate  of  individual  connection  to  converge  to  an  equilibrium  manifold  containing  the 
desired  equilibrium.  On  the  slow  timescale,  the  proposed  solution  applies  HMD  control  law 
to  control  the  number  of  TCP  connections,  in  order  to  converge  to  the  desired  equilibrium 
along  the  equilibrium  manifold.  Eventually,  the  sending  rates  controlled  by  the  proposed 
solution  converge  to  the  desired  equilibrium  exponentially  fast,  as  expected. 

By  always  using  TCP  as  the  control  law  for  sending  rate  in  fast  timescale,  and  designing 
new  control  law  for  number  of  connections  in  slow  timescale,  one  can  potentially  address 
any  flow  control  problem  without  changing  today’s  transport  layer  protocol.  Further,  if  the 
control  law  for  the  number  of  connections  utilizes  information  that  today’s  infrastructure 
can  readily  provide,  then  the  problem  is  solved  without  modifying  network  infrastructure 
as  well. 

One  significant  advantage  of  this  two  timescale  framework  is  the  separation  of  control 
laws  in  two  timescales.  Control  law  in  each  timescale  can  be  designed  independently  of  the 
control  law  in  the  other  timescale.  As  long  as  the  control  law  can  guarantee  exponential 
convergence  in  its  timescale,  the  general  exponential  convergence  result  holds.  This  implies 
that  we  can  replace  one  control  law  in  one  timescale,  without  affecting  the  one  in  the 
other  timescale,  and  the  general  convergence  result  still  holds.  For  example,  as  argued  in 
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previous  chapter,  we  can  use  TFRC  rather  than  TCP  to  control  sending  rate  of  individual 
connections,  and  still  achieve  all  convergence  properties.  This  is  analogous  to  upgrading  a 
car  by  changing  the  tires  without  changing  the  engine. 


3.7  A  Variant  to  Proposed  Solution 


In  this  section,  we  describe  a  variant  to  proposed  solution,  by  using  a  different  control 
law  for  the  number  of  connections  in  a  slow  timescale.  This  flexible  design  shows  the  power 
of  the  general  two  timescale  framework. 


The  variant  solution  uses  TCP  to  control  sending  rate  of  individual  connection,  and  uses 
Inversely  Increase  and  Additively  Decrease  (HAD)  to  control  the  number  of  connections. 
That  is,  the  number  of  connections  increases  inverse  proportionally  upon  no  congestion, 
and  decreases  additively  upon  congestion.  The  only  difference  between  this  variant  solution 
and  proposed  solution  is  the  control  law  for  the  number  of  connections,  i.e.  HAD  vs  HMD. 
The  approximated  version  of  the  variant  solution  is  the  following: 

*r(0  =  2SrT(t)  (25y}  -  XrW  ^jer iej  +  9j(Vj (*))])  , 

<  /  \  (3.24) 

nr(t)  =  Cr  ^  -  f{Ejer9j(yj(t)))  • 


Applying  two  timescale  decomposition,  we  derive  the  boundary  layer  system  in  fast 
timescale: 

_  1  (  2 S2nl{t)  _  2 1 


Xr{t)  =  2 SliRT  (  T't  -  ^j€r[ej  +  9j(Vj(t 
nr(t)  =  constant, 
and  the  reduced  order  system  in  slow  timescale: 

nr(t)y/2S 


Xr(t)  = 


Tr^/T,jerlej+9j(yj(t))]  ' 


Mt)  =Cr[^)-  f(Ej&9j(Vj(t))) 


(3.25) 


(3.26) 


Again,  the  question  here  is  whether  the  variant  solution  leads  to  a  system  with  a  unique 
exponential  stable  equilibrium,  at  which  all  flow  control  goals  are  achieved.  Following  similar 
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analysis  of  proposed  solution,  we  have  the  following  theorem  for  existence  and  uniqueness 
of  the  optimal  equilibrium: 


Theorem  3.7.1.  For  arbitrary  (5  >  0,  the  approximate  system  in  Equation  (3.24)  has  a 
unique  equilibrium,  denoted  by  (x*,n*)  as 


/!  >: y,  r'-hO:)] )) '  rGjR; 


XZ  = 


V2S 


(3.27) 


r,  r  €  R. 


/(Ejer  9j  {y*j))Tr  \/T,jer[9j{yj)+£j\ 

Further,  this  unique  equilibrium  solves  the  following  concave  optimization  problem 


Vi 


max 

x>0 


E  Ur(X r)  -  E  /  9j{z)  dz, 


reR 


leJ’ 


with  Ur  being  concave  function: 


fXr  (  25 2 

Ur(xr)  =  I  hf1  l  ^2^2  )du,  reR, 


where  hr  1  is  t/ie  inverse  of  a  monotonically  increasing  function  hr: 


edz  _  1 

e?z  +  1 


hr(z)  ~  E  ei  +  z)  =  (  E  + 


j&r 


(3.28) 


(3.29) 


reR. 


Proof.  Refer  to  Appendix  G.  □ 

At  the  equilibrium,  as  compared  to  proposed  solution,  all  observations  about  xr(t)  are 
the  same,  except  a  slightly  different  net  utility  shown  in  Equation  (3.29)  is  maximized. 

Since  only  the  reduced  order  system  is  modified,  according  to  the  general  two  timescale 
framework,  we  need  to  show  reduced  order  system  has  an  exponentially  stable  equilibrium, 
in  order  to  show  the  entire  system  converges  to  the  desired  equilibrium  exponentially  fast. 
The  following  theorem  shows  the  exponential  stability  of  the  equilibrium  of  reduced  order 
system: 
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Theorem  3.7.2.  For  arbitrary  (3  >  0,  the  unique  equilibrium  of  the  reduced  system  in 


Equation  (3.26)  is 

•  locally  exponentially  stable, 

•  globally  asymptotically  stable, 

•  semi  globally  exponentially  stable. 


Proof.  For  local  exponential  stability,  the  proof  follows  exactly  the  same  idea,  technique, 
and  procedure  of  the  proof  of  Theorem  3.3.2  in  Appendix  B,  with  the  exception  of  few 
equations.  So  we  will  not  go  over  the  details  here. 


For  globally  asymptotically  stability,  the  proof  follows  exactly  the  same  idea,  technique, 
and  procedure  of  the  proof  of  Lemma  3.3.3  in  Appendix  D,  except  for  using  a  different  La 
Salle  function  in  the  proof,  as  follows: 


V{n,z)  = 


r£R 


y/yrir 


fZr  j 

dy  ~  /  —f(y  -  er)dy 

■hr  Vv 


+ 


1  ,  ,1 


<t>r(~)dy 


(3.30) 


Jo  r  y 

where  function  <fr{-)  is  defined  in  Equation  (D.2).  So  we  will  not  go  over  the  details  here. 


Applying  Lemma  3.3.4,  we  get  the  desired  semi  globally  exponential  stability  result.  □ 


Similar  to  the  proposed  solution  in  Equation  (3.14),  in  the  variant  solution  shown  in 
Equation  (3.24),  cr  can  be  chosen  in  a  distributed  fashion,  as  long  as  the  two  timescale 
assumption  holds.  Practically,  this  implies  that  each  user  can  adjust  nr(t)  according  to 
the  same  control  law  but  with  a  different  rate.  Specifically,  a  global  setting  among  all  the 
users  is  not  necessary.  Furthermore,  allowing  some  of  the  cr  to  be  zero  represents  a  scenario 
according  to  which  the  proposed  scheme  coexists  with  TCP.  In  this  situation,  all  theorems 
still  hold,  except  for  a  modification  to  Theorem  3.7.1.  More  precisely,  we  have  the  following 
Corollary: 
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Corollary  3.7.1.  For  arbitrary  topology,  arbitrary  number  of  users  running  either  TCP 
or  the  variant  solution  shown  in  Equation  (3.24),  and  arbitrary  initial  sending  rates,  the 
following  holds: 


Sending  rates  x(t)  converge  to  a  unique  equilibrium: 


—  Users  running  TCP: 


x„  = 


y/2. S 


9j  [y*i)  +£j 


(3.31) 


Tr\J 

-  Users  running  the  variant  solution  as  in  Equation  (3.24): 
*  v/2S 


=  (3.32) 

f('Ej&9j(Vj))Tr^'Ej&  [dj  ( y*j )  +ei 

At  the  equilibrium,  all  bottlenecks  are  fully  utilized  for  large  [3,  and  the  following 
concave  optimization  problem  is  solved: 


Vi 


max 

x>0 
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with  Ur  being  concave  function: 


(3.33) 


Users  running  TCP: 


Ur(xr  )  —  ^  ' 
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Users  running  the  variant  solution  as  in  Equation  (3.24): 


f  Xr  (  2  S2 

Ur(xr)  =  /  hfl  (  )  du, 


where  hr  1  is  the  inverse  of  an  monotonically  increasing  function  hr: 


K{z)  =  (  £  ei  +  z j  /2(*)  =  f  £  ei  +  z 
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Similar  to  proposed  solution  coexisting  with  TCP,  an  observation  on  the  fairness  among 
users  is  that  at  the  equilibrium,  users  running  the  variant  solution  are  fair  to  users  running 
TCP  in  the  following  sense: 

•  In  the  case  that  TCP  operates  in  capacity-limited  region,  i.e.  all  bottleneck  links  are 
fully  utilized  even  if  all  users  running  proposed  solution  only  open  one  connection,  for 
large  values  of  /?,  the  approximated  indicator  function  /(•)  takes  the  value  1.  Users 
running  the  variant  solution  will  open  only  one  connection,  and  their  sending  rates 
will  converge  to  the  same  value  as  if  they  were  running  TCP.  In  this  case,  fairness 
among  users  running  TCP  and  the  variant  solution  is  the  same  as  fairness  among 
TCP  users. 

•  In  the  case  that  TCP  operates  in  channel-error-limited  region,  i.e.  the  bottleneck 
links  of  some  routers  are  underutilized,  if  users  running  the  variant  solution  open 
one  connection,  for  large  values  of  (3,  approximated  indication  function  /(•)  takes  on  a 
value  between  0  and  1,  and  approximated  congestion  loss  function  g{  )  takes  on  a  very 
small  positive  value,  as  seen  from  definition  of  /(•)  in  Equation  3.13.  Consequently, 
as  seen  from  users’  equilibrium  rates  in  the  above  Corollary,  users  running  the  variant 
solution  will  get  the  residual  bandwidth  without  affecting  TCP  users’s  throughput, 
i.e.  users  running  TCP  get  throughput  close  to  what  they  would  have  gotten  if  no 
other  users  were  running  the  proposed  solution. 
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Chapter  4 


Proposed  Practical  Solutions 


In  this  chapter,  we  design  practical  solutions  for  flow  control  in  wireless  networks,  fol¬ 
lowing  the  insights  gained  from  analysis  in  the  previous  Chapter.  In  particular,  we  will 
first  discuss  the  guidelines  for  implementing  the  proposed  solution  in  Equations  (3.9)  and 
(3.10)  in  practice.  Then  we  design  two  practical  solutions  following  the  guidelines,  namely 
Enhanced  Multiple  TCP  (E-MULTCP)  for  data  transmission  and  Enhance  Multiple  TFRC 
(E-MULTFRC)  for  multimedia  streaming.  We  discuss  how  the  parameters  in  E-MULTCP 
and  E-MULTFRC  are  chosen  to  achieve  the  desired  performance,  and  to  balance  band¬ 
width  utilization  and  responsiveness  to  changes  in  network  conditions.  We  also  discuss  the 
quantization  drawback  of  E-MULTFRC  related  to  operating  multiple  connections,  and  de¬ 
sign  a  variant  called  Enhanced  ALL-IN-ONE  TFRC  (E-AIOTFRC)  to  address  it.  Results 
of  NS-2  simulations  and  actual  experiments  are  provided  in  Chapter  5  characterizing  the 
performance  of  all  above  schemes. 
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Table  4.1.  One-to-one  correspondence  between  congestion  status  and  queuing  delay. 


Status 

Queuing  Delay 

Congested 

Not  Congested 

>  0 

=  0 

4.1  Design  Guideline 

We  conceptually  rewrite  the  proposed  solution  in  Equations  (3.9)  and  (3.10)  as  follows: 
xr(t)  =  nr(t)  connections  of  TCP, 

< 

hr(t)  =  IIMD  : 

Above  equations  show  design  guidelines  of  practical  schemes: 

•  Use  TCP  to  control  sending  rate  of  individual  connection,  based  on  observed  overall 
end-to-end  packet  loss  rates; 

•  On  a  timescale  much  slower  than  the  one  at  which  TCP  operates,  control  the  number 
of  TCP  connections  according  to  IIMD  control  law,  based  on  additional  one  bit  of 
information  indicating  whether  user’s  route  is  congested  or  not. 

Since  TCP  operates  on  a  timescale  from  milliseconds  to  seconds,  we  choose  to  adjust  the 
number  of  connections  on  the  order  of  tens  of  seconds,  in  order  to  satisfy  the  two  timescale 
assumption. 

One  bit  of  information  indicating  congestion  status  of  user’s  route  is  also  needed  to 
design  practical  schemes.  To  get  the  information  from  end-to-end  measurement,  it  is  well- 
known  that  there  is  one-to-one  correspondence  between  congestion  status  of  a  route  and 
the  positivity  of  queuing  delay  along  it,  as  shown  in  Table  4.1.  Hence,  we  can  measure  an 
average  queuing  delay  in  the  interval,  over  which  the  number  of  connections  is  adjusted. 

As  nr(t)  shown  in  Equation  (4.1)  is  not  required  to  be  an  integer,  in  practice  one  can 
either  quantize  it  to  the  closest  integer,  or  mimic  sending  rate  of  nr(t)  connections  by 
opening  one  TFRC  connection  with  rate  equal  to  that  of  nr(t)  connections.  Alternatively, 
it  is  possible  to  open  an  integer  number  of  connections  with  varying  packet  sizes,  such  that 


cA^jt )  ~  nr{t)]i  ^  route  r  is  congested, 
otherwise. 


(4.1) 


nr(t)  - 
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the  aggregate  sending  rate  is  the  same  as  that  of  nr(t)  connections  with  standard  packet 
size.  For  example,  if  we  want  to  mimic  sending  rate  of  1.5  connections  with  standard  packet 
size,  we  can  open  2  connections  with  packet  size  chosen  to  be  0.75  of  the  standard  one. 

Following  these  guidelines,  we  design  practical  schemes  for  data  transmission,  as  well  as 
for  multimedia  streaming,  in  wireless  networks. 


4.2  E-MULTCP  for  Data  Transmission 

The  framework  of  our  proposed  E-MULTCP  is  shown  in  Figure  4.1.  As  seen,  there  are 
two  components  in  the  system:  RTT  Measurement  Subsystem  (RMS),  and  Connections 
Controller  Subsystem  (CCS). 


Figure  4.1.  E-MULTCP  system  framework. 


4.2.1  RMS 

The  gray  blocks  in  Figure  4.1  represent  RMS.  They  measure  round  trip  time  samples 
between  sender  and  receiver,  denoted  by  rttsampie ,  by  computing  difference  between  the 
time  sender  emits  a  packet,  and  the  time  it  receives  the  ACK  from  receiver.  These  packets 
are  sent  through  the  forward  and  backward  UDP  connections  between  sender  and  receiver. 
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To  reduce  overhead  and  to  prevent  congestion  collapse,  the  rate  at  which  RTT- 
measurement  packets  are  sent,  is  set  to  be  the  same  as  that  of  data  packets,  which  is 
estimated  on  sender  side  by  measuring  the  number  of  data  packets  sent  in  the  previous 
round  trip  time  interval.  The  RTT  measurement  packets  contain  only  an  IP  header  and 
a  timestamp,  a  total  of  28  bytes.  In  the  case  when  TCP  uses  typical  packet  size  of  1500 
bytes,  the  RTT  measurement  overhead  is  2%  of  the  total  throughput. 

After  waiting  for  a  fixed  interval,  denoted  by  r,  RMS  computes  the  running  average, 
avejrtt,  of  these  rttsampie s,  and  reports  it  to  the  CCS.  As  such,  1/r  is  the  frequency  of 
adjusting  number  of  connections;  r  has  to  be  large  enough  to  ensure  the  frequency  is  much 
lower  than  that  of  changing  the  source  rate,  which  is  typically  of  the  order  of  round  trip 
time.  In  our  current  implementation,  we  choose  r  to  be  20  seconds.  It  should  be  noted  here 
that  our  formulation  in  Section  3.4  allows  variable  settings  of  r,  rather  than  a  fixed  value, 
such  as  20  seconds. 

4.2.2  CCS 

The  CCS  is  shown  as  the  white  blocks  in  Figure  4.1.  Its  basic  functionality  is  to  properly 
control  the  number  of  connections  n,  following  the  control  law  shown  in  Equation  (4.1).  CCS 
at  the  sender  adjusts  the  number  of  connections  n  roughly  by  HMD  law,  based  on  route 
congestion  status.  Specifically,  avejrtt  is  reported  to  CCS  by  RMS,  CCS  sets  the  rttjmin 
as  the  minimum  over  all  avejrtt  seen  so  far,  and  then  adapts  the  number  of  connections  n 
as  follows: 

Bn  +  a/n,  if  avejrtt  —  rttjmin  >  - yrttjmin ; 
n=l  (4.2) 

n  +  a/n,  otherwise. 

where  a  =  1  —  (3  <  1  and  7  is  a  preset  parameter.  This  is  nothing  but  a  discrete  implemen¬ 
tation  of  the  HMD  control  law  shown  in  Equation  (4.1),  with  cr  =  a/r.  The  congestion 
status  is  estimated  by  comparing  the  measured  queuing  delay  avejrtt  —  rttjmin  with  a 
dynamic  threshold  7 rttjmin .  For  a  given  route,  the  rttjmin  is  a  constant  representing 
the  minimum  observed  round  trip  time  for  that  route,  approximating  physical  propagation 
delay.  As  such,  avejrtt  —  rttjmin  corresponds  to  current  queuing  delay,  and  'yrttjmin  is  a 
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threshold  on  the  queuing  delay  that  E-MULTCP  can  tolerate  before  it  starts  to  decrease 
the  number  of  connections.  Therefore,  under  ideal  conditions,  E-MULTCP  keeps  increasing 
the  number  of  connections  to  make  avejrtt  as  close  as  possible  to  (1  +  7 )rttjnin  without 
exceeding  it. 


4.2.3  Discussion 


In  the  increasing  stage,  i.e.  I(ave^rtt  —  rtt  min  >  7 rtt-min )  =  0,  E-MULTCP  has 
an  increasing  rate  of  h(t)  =  a/(rn{t))  according  to  Equation  3.10.  Hence,  to  increase  n 
from  N2  to  Ni,  assuming  N±  >  N2,  it  roughly  takes  E-MULTCP  ^(IV2  —  IVf)  seconds. 
a  is  defined  as  the  normalized  increasing  rate,  i.e.  the  increasing  rate  h(t)  normalized  by 
rn(t).  It  is  a  crucial  design  parameter  independent  of  n(t)  and  r,  representing  how  fast 
E-MULTCP  shifts  from  channel-error-limited  region  to  capacity-limited  region.  The  larger 
a  is,  the  faster  E-MULTCP  increases  its  overall  rate. 

The  decreasing  rate  for  the  decreasing  stage  is  roughly  h(t)  =  — an^ ,  and  hence  it 
takes  E-MULTCP  \n(N\/N2)  seconds  to  decrease  n  from  N\  to  A^.  Thus,  E-MULTCP 
is  conservative  in  adding  connections,  but  much  aggressive  in  closing  them. 


In  a  simple  topology  with  one  E-MULTCP  system  over  one  wireless  link,  we  can  use 
the  above  results  to  compute  bandwidth  utilization  ratio.  At  steady  state,  E-MULTCP 
periodically  increases  the  number  of  connections  n  to  the  optimal  n*  that  fully  utilizes  the 
wireless  bandwidth,  and  proportionally  decreases  n  upon  reaching  n*.  The  approximated 
continuous  version  of  this  process  is  demonstrated  in  Figure  4.2.  In  the  plot,  T  is  the 
time  for  n(t)  to  increases  from  /3n*  to  n*,  and  hence  is  (n*)2(  1  —  /32)r /2a.  The  number  of 
connections  is  given  by  n{t)  =  y/(t  —  kT)2a/r  +  (/ 3n *)2  for  kT  <  t  <  (k+l)T  and  k  £  Z+ . 


The  utilization  ratio,  at  steady  state,  is  then  the  ratio  between  the  average  number  of 
connections  and  n* .  as  follows: 


1 


r1  2 

Jo  n{8)d5  =  - 


21  +  /3  +  /32 


(4.3) 


Tn*  J o  "  3  1  •  J 

This  ratio  is  at  least  2/3,  only  depends  on  /?,  and  is  independent  of  wireless  packet  loss  rate, 
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n* 


Figure  4.2.  Demonstration  of  the  change  on  the  number  of  connections  n,  controlled  by 
single  E-MULTCP  over  single  wireless  link. 

r,  and  n*.  The  larger  f3  is,  the  high  utilization  ratio  is.  There  is  clearly  a  trade-off  between 
above  utilization  ratio  and  normalized  increasing  rate,  i.e.  a  =  1  —  f3,  as  shown  in  Figure 
4.3.  It  is  up  to  designer  to  select  /3  to  get  the  best  balance  for  targeting  scenario. 

One  of  our  main  target  scenarios  for  E-MULTCP  is  cell  phones  connecting  to  Internet 
via  commercial  wireless  networks  and  downloading  data  or  video  from  servers.  As  described 
later,  we  have  empirically  found  that  2  or  3  parallel  connections  are  sufficient  to  fully  utilized 
lxRTT  CDMA  and  EVDO  wireless  bandwidth,  which  are  around  110  kbps  and  380  kbps, 
respectively  [16].  As  the  wireless  bandwidth  is  a  scarce  resource  and  limited  in  nature, 
and  the  optimal  number  of  connections  n*  in  most  practical  situations  is  small,  it  is  more 
important  to  achieve  high  utilization  than  high  increasing  rate.  Moreover,  most  cell  phones 
are  limited  in  power,  so  it  would  be  better  if  (3  is  selected  to  make  E-MULTCP  control 
procedure  computationally  efficient,  by  replacing  multiplication  operations  by  binary  shifts. 

As  such,  we  have  selected  j3  =  0.75  for  E-MULTCP,  to  get  a  utilization  ratio  of  0.88, 
and  a  normalized  increasing  rate  of  0.25.  This  means  it  takes  E-MULTCP  2r(Nf  —  N%) 
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seconds  to  increase  from  N2  connections  to  N±  connections.  The  HMD  of  n  requires  binary 
shifts  and  additions  only,  and  is  therefore  computationally  efficient. 


Figure  4.3.  A  trade  off  between  utilization  ratio  and  normalized  increasing  rate  a. 

When  there  is  a  route  change  either  due  to  change  in  the  wireless  base  station,  or  due  to 
route  change  within  the  wired  networks,  the  value  of  rttjnin  changes  significantly,  affecting 
the  performance  of  E-MULTCP.  Under  these  conditions,  it  is  conceivable  to  use  route  change 
detection  tools  such  as  traceroute  [5]  at  the  sender  to  detect  the  route  change,  in  order  to 
reset  rtt,_min  to  a  new  value.  Furthermore,  it  can  be  argued  that  the  overall  throughput  of 
E-MULTCP  will  not  go  to  zero,  resulting  in  starvation;  this  is  because  E-MULTCP  always 
keeps  at  least  one  connection  open. 

Since  the  data  stream  in  E-MULTCP  is  transmitted  using  multiple  connections,  the 
receiver  could  potentially  receive  out  of  order  packets.  However,  reordering  application 
packets  from  multiple  TCP  connections  using  a  receive  buffer  is  a  rather  mature  technology 
widely  used  in  peer-to-peer  applications,  such  as  Bit  Torrent  file  sharing,  e.g.  Kazza  [2], 
and  peer-to-peer  streaming,  e.g.  PPlive  [4],  As  discussed  in  later  chapters,  we  have  also 
implemented  and  successfully  tested  a  file  downloading  software  on  an  actual  BREW  [1] 
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cellular  telephone,  combining  E-MULTCP  and  a  simple  packets  reordering  procedure.  We 
refer  interested  readers  to  details  of  reordering  technology  in  literature  [59] . 

In  summary,  in  the  E-MULTCP  system,  the  number  of  connections  is  controlled  ac¬ 
cording  to  a  discrete  version  of  Equation  (3.10)  at  the  sender;  the  sending  rate  of  each  TCP 
connection  is  adjusted  automatically  by  itself.  The  rate  of  change  of  number  of  connections 
is  expected  to  be  much  slower  than  that  of  sending  rate  satisfying  the  two  time  scale  as¬ 
sumption  in  the  previous  section.  Thus,  the  optimality  and  stability  analysis  in  the  previous 
section  for  the  dynamic  system  in  Equations  (3.9)  and  (3.10)  applies  to  E-MULTCP,  indi¬ 
cating  that  E-MULTCP  results  in  a  stable,  yet  fully  utilized  network  as  shown  in  Equation 
(3.20). 

4.3  E-MULTFRC  for  Multimedia  Streaming 

Although  the  analysis  in  previous  section  is  based  on  TCP  model,  it  can  be  extended  to 
TFRC,  since  it  has  been  shown  TFRC  has  the  same  stationary  behavior  as  TCP  [23].  As 
we  are  interested  in  rate  control  for  streaming  over  wireless,  we  design  a  practical  scheme 
for  that,  based  on  our  analysis  and  control  law  in  Equation  (4.1),  by  adjusting  the  number 
of  TFRC  connections. 

The  framework  of  our  proposed  system  which  we  refer  to  as  E-MULTFRC  is  shown  in 
Figure  4.4.  As  seen,  the  system  is  quite  similar  to  that  of  E-MULTCP  shown  in  Figure 
4.1.  There  are  two  components  in  the  system:  RTT  measurement  subsystem  (RMS),  and 
connections  controller  subsystem  (CCS). 

4.3.1  RMS 

The  gray  block  in  Figure  4.4  represents  RMS  that  resides  at  the  sender;  it  receives 
reports  from  receiver  every  round  trip  time,  containing  average  round  trip  time  rttsampie 
measured  in  the  past  round  trip  time  window.  After  waiting  for  a  fixed  interval,  denoted 
by  r,  RMS  computes  the  running  average,  avejrtt,  of  these  rttsampie s,  and  reports  it  to  the 
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Figure  4.4.  E-MULTFRC  system  framework. 

CCS.  Here,  1/t  defines  the  frequency  of  adjusting  number  of  connections;  it  has  to  be  large 
enough  to  ensure  the  frequency  is  much  lower  than  that  of  changing  the  source  rate,  which 
is  typically  of  the  order  of  round  trip  time.  In  our  current  implementation,  we  choose  r  to 
be  20  seconds. 

4.3.2  CCS 

The  CCS  subsystem  in  E-MULTFRC  is  the  same  as  the  one  in  E-MULTCP,  hence  we 
will  not  repeat  the  details  here. 


4.4  Multiple  TFRC  (MULTFRC)  for  Multimedia  Streaming: 
A  Variant  of  E-MULTFRC 

As  seen  in  previous  section,  E-MULTFRC  follows  the  proposed  solution  in  Chapter  3. 
In  this  section,  we  develop  a  scheme  called  MULTFRC  that  is  a  variant  of  proposed  solution, 
as  shown  in  Section  3.7,  and  a  variant  of  E-MULTFRC. 

MULTFRC  shares  the  same  design  principles  as  E-MULTFRC,  expect  that  it  applies 
HAD  instead  of  HMD  to  control  the  number  of  TFRC  connections.  That  is,  MULTFRC  also 
has  a  RMS  and  a  CCS  subsystem,  but  the  CCS  system  controls  the  number  of  connections 
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n,  as  follows: 


n  —  1 


if  avejrtt  —  rtt,_min  >  7 rttjmin] 


n  =  < 


n  +  a/n,  otherwise. 


(4.4) 


Similar  to  the  analysis  in  Section  4.2.3,  adaptation  rate  and  utilization  of  MULTFRC 
in  a  simple  topology  with  one  MULTCP  system  over  one  wireless  link  can  be  analyzed  as 
follows. 


In  the  increasing  stage,  MULTFRC  has  the  same  increasing  rate  as  E-MULTFRC,  as 
they  share  the  same  increase  law  on  n.  For  example,  to  increase  the  number  of  connections 
n  from  N2  to  N\,  assuming  N\  >  N2,  it  roughly  takes  MULTFRC  ^rjNf  —  N‘$)  seconds. 
On  the  other  hand,  the  decreasing  rate  of  MULTFRC  for  the  decreasing  stage  is  roughly 
h(t)  =  —  y,  and  hence  it  takes  MULTFRC  t(N\  —  N2)  seconds  to  decrease  n  from  N\  to 
N2. 


The  utilization  ratio  of  MULTFRC,  at  steady  state,  is  then  the  ratio  between  the  average 
number  of  connections  and  n* .  Note  increasing  from  n*  —  1  to  n*  takes  T  =  r(2n*  —  1)/ (2cc), 
and  n(t )  =  y/2[at/r  +  n2(0)]  in  the  increasing  stage,  the  utilization  ratio  can  be  computed 
as  follows: 


Tn* 


n(5)d5  =  -  1  — 


1  3n*  —  2 
n*  6 n*  —  3 


>  1  - 


2  n* 


(4.5) 


This  ratio  is  at  least  (2/3) 2 ,  only  depends  on  n*,  and  is  independent  of  a  and  r.  As  n*  — >  00, 
the  utilization  ratio  tends  to  be  1.  Hence,  large  values  of  n*  improve  the  utilization  ratio 
in  this  scenario.  As  large  values  of  n*  typically  represent  poor  wireless  channel  conditions, 
theoretically  MULTFRC  has  better  utilization  ratio  as  the  channel  gets  worse. 


4.5  E-AIOTFRC  for  Multimedia  Streaming 

There  are  mainly  two  drawbacks  associated  with  E-MULTFRC.  First  drawback  has  to 
do  with  bandwidth  underutilization,  and  the  other  two  are  with  implementation  complexity. 
We  will  begin  with  utilization  drawback.  As  we  will  show  in  the  next  chapter,  E-MULTFRC 
may  not  fully  utilize  the  wireless  bandwidth  when  wireless  channel  error  rate  is  small.  This 
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suboptimal  performance  has  two  causes.  First  one  is  the  control  behavior  described  in 
Equation  (4.4):  as  described,  n  is  decreased  when  the  full  utilization  of  bottlenecks  is 
detected,  and  is  inversely  increased  until  the  next  full  utilization  is  detected.  During  this 
period,  bottlenecks  stay  underutilized,  resulting  in  suboptimal  average  throughput.  It  is 
impossible  to  remove  this  sub-optimality  determined  by  the  control  law  without  changing 
the  law.  The  second  reason  for  bandwidth  underutilization  is  the  “quantization  effect”  in 
E-MULTFRC  whereby  in  practice  the  number  of  connections  is  forced  to  be  an  integer. 
This  loss  of  granularity  typically  results  in  bandwidth  underutilization.  For  example,  if  the 
optimal  number  of  connections  has  been  determined  to  be  1.5,  then  n  is  forced  to  take 
fractional  values  between  1  and  3,  e.g.  1,  1.25,  1.45,  2.14,  1.14,  ...,  as  dictated  by  Equation 
(4.4).  E-MULTFRC  then  quantizes  n  to  the  closest  integer  to  oscillate  between  one  and 
two,  resulting  in  loss  of  throughput  granularity.  This  effect  can  be  eliminated  by  avoiding 
the  quantization  step,  as  described  below. 

The  second  drawback  of  E-MULTFRC  is  of  a  more  practical  nature.  Operating  multiple 
connections  in  one  application  could  potentially  consume  too  much  system  resources.  For 
example,  each  TFRC  connection  uses  a  different  port  to  send  out  data  packets,  carries  out 
individual  feedback  process,  and  updates  the  loss  event  rate  and  RTT  even  though  they 
are  highly  correlated  for  these  TFRC  connections.  Clearly,  there  is  unnecessary  overhead 
associated  with  operating  multiple  connections,  in  terms  of  computation,  processing  power, 
memory,  and  ports,  particularly  for  today’s  low  power,  resource-limited  handheld  devices. 

In  this  section  we  propose  a  new  scheme  E-AIOTFRC,  in  order  to  address  these  draw¬ 
backs,  while  retaining  the  same  control  law  for  n  as  in  E-MULTFRC.  To  achieve  this 
goal,  we  integrate  the  Bandwidth  Filtered  Loss  Detection  (BFLD)  technique  from  [39],  to 
be  described  shortly,  together  with  the  control  law  in  Equation  (4.1)  to  construct  the  E- 
AIOTFRC  system.  The  system  framework  is  shown  in  Figure  4.5.  Basically,  the  Sink  at 
the  receiver  feeds  back  the  RTT  and  loss  event  rate  to  the  sender.  The  sender  then  adjusts 
of  n  based  on  Equation  (4.1),  and  sends  out  the  data  packets  at  a  rate  of  n  times  that 
of  one  TFRC’s  sending  rate.  The  functionalities  of  E-AIOTFRC  senders  and  receivers  are 
described  as  follows: 
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Figure  4.5.  The  system  framework  of  E-AIOTFRC. 

•  Sender :  There  are  two  functional  components  in  the  sender.  One  component  is  rep¬ 
resented  by  the  “compute  n"  block.  It  receives  the  RTTs  from  the  receiver,  computes 
an  avejrtt,  by  averaging  these  RTT  samples  over  a  20  second  window,  then  updates  n 
according  to  Equation  (4.4)  every  20  seconds,  i.e.  using  the  same  law  as  E-MULTFRC. 
For  E-AIOTFRC,  we  choose  a  =  1  —  f3  =  0.25,  and  7  =  0.5. 

The  other  component  is  represented  by  the  “TFRC+BFLD”  block,  and  it  has  two 
functionalities:  first,  it  obtains  the  updated  n  from  the  “compute  n”  component,  as 
well  as  the  loss  event  rate,  denoted  as  pi  from  the  receiver.  It  then  computes  the 
TCP  friendly  rate  of  one  TFRC  connection  as  the  standard  TFRC  does  [23],  which 
is  roughly  proportional  to  &  being  a  constant  and  Tr  being  the  measured 

round  trip  time,  and  adjusts  the  sending  rate  to  be  n  times  that  of  one  TFRC. 

The  second  functionality  of  the  “TFRC+BFLD”  block  in  the  sender  is  to  mark  the 
headers  of  selected  data  packets  before  they  are  sent  out.  The  data  packets  to  mark  are 
selected  in  such  a  way  that  they  form  a  virtual  single  TFRC  flow,  and  hence  correspond 
to  1/n  of  all  the  outgoing  packets.  For  example,  if  n  =  1.5,  then  “TFRC+BFLD” 
evenly  marks  2/3  of  all  outgoing  packets.  The  reason  for  the  marking  is  to  facilitate 
the  loss  event  rate  measurement  at  the  receiver,  and  will  be  explained  shortly. 

•  Receiver:  The  E-AIOTFRC  Sink  component  reports  the  RTT  and  the  loss  event 
rate  of  the  virtual  TFRC  connection  to  the  sender  every  RTT.  The  only  difference 
between  the  E-AIOTFRC  Sink  and  the  original  TFRC  Sink  is  that  the  E-AIOTFRC 
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Sink  measures  and  updates  the  loss  event  rate  based  on  the  virtual  TFRC  flow  with 
marked  packets. 

The  operation  flow  of  E-AIOTFRC  is  as  follows.  Every  RTT,  the  receiver  sends  back 
the  measured  RTT  and  loss  event  rate  to  the  sender.  Based  on  the  RTT,  the  sender 
adjusts  n  according  to  Equation  (4.4)  every  20  seconds.  At  any  moment,  the  sender 
sends  at  a  rate  equivalent  to  n  TFRC  flow  shares,  and  marks  the  selected  outgoing 
packets  to  form  a  virtual  stream,  which  is  used  for  the  receiver  to  carry  out  loss  event 
rate  measurements. 

It  should  also  noted  that  all  the  functionalities  of  sender  can  be  shifted  to  and  imple¬ 
mented  at  receiver: 

•  Virtual  sampling  on  sender  side  can  be  done  instead  at  receiver  by  taking  only  1/n  of 
all  incoming  and  lost  packets  into  loss  event  rate  measurement  and  update. 

•  Based  on  the  measured  round  trip  time,  receiver  can  carry  out  the  adjustment  of  n, 
discount  the  measured  packet  loss  event  rate  pi,  by  1/n2,  and  send  this  discounted 
packet  loss  event  rate  pi/n 2  to  sender.  Using  this  discount  packet  loss  event  rate  and 
the  measured  round  trip  time  Tr,  sender  computes  the  source  rate  as  Tn^,  equivalent 
to  that  of  n  TFRC  connections.  Hence,  the  sender  will  send  out  the  packets  at  a  rate 
equivalent  to  n  TFRC  connections. 

By  shifting  all  functionalities  to  receiver,  it  is  possible  to  implement  E-AIOTFRC  scheme  by 
only  modifying  the  receiver,  i.e.  the  client  side  which  is  typically  a  PC,  laptop  or  a  handheld 
device.  This  implementation  requires  no  modification  at  the  sender,  i.e.  the  server  side, 
potentially  making  E-AIOTFRC  even  easier  to  deploy. 

We  now  explain  the  reasoning  behind  the  marking  process.  It  has  been  argued  in  [39] 
that  if  the  sending  rate  of  the  application  is  adjusted  to  be  n  times  that  of  one  TFRC,  then 
TFRC  Sink  at  the  receiver  underestimates  the  loss  event  rate  based  on  using  all  the  received 
packets.  TFRC  Sink  records  the  beginning  of  a  loss  event  when  a  packet  loss  is  detected. 
The  loss  event  ends  when,  after  a  “guarding”  period  of  one  RTT,  another  packet  loss  is 
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detected.  A  loss  event  interval  is  defined  as  the  difference  in  sequence  numbers  between 
the  above  two  lost  packets;  the  loss  event  rate  is  thus  estimated  by  taking  the  inverse  of 
this  difference.  In  the  case  where  all  the  packets  are  used  to  estimate  loss  event  rate,  the 
number  of  packets  received  by  TFRC  Sink,  during  the  guarding  RTT,  is  n  times  that  of 
one  TFRC.  As  such,  the  loss  event  interval  now  could  be  n  times  of  that  of  single  TFRC, 
and  the  loss  event  rate  is  underestimated. 

To  overcome  this  problem,  we  sample  the  outgoing  packets  at  the  sender  to  form  a  single 
virtual  TFRC  stream  at  the  sender,  and  modify  TFRC  Sink  to  carry  out  the  loss  event  rate 
measurement  based  on  this  virtual  stream.  This  is  the  functionality  of  BFLD  as  verified  in 
[39]. 

If  otherwise  the  deflated  loss  event  rate  is  reported  to  the  sender,  the  aggregate  sending 
rate  will  in  fact  be  higher  than  n  TFRCs’  flow  shares.  This  is  because  the  aggregate  sending 
rate  is  inversely  proportional  to  the  square  root  of  its  measured  loss  event  rate,  which 
is  smaller  than  those  measured  by  individual  TFRC  flow.  Consequently,  this  aggressive 
sending  rate  could  potentially  cause  congestion  collapse  or  unfairness  to  TCP.  This  is  also 
of  concern  to  the  MULTFRC-LERD  scheme  in  [52], 

We  now  explain  the  reason  to  choose  a  larger  7  in  E-AIOTFRC  than  in  E-MULTFRC. 
E-MULTFRC  achieves  n  TFRC  flow  shares  by  opening  n  independent  TFRC  connections, 
while  E-AIOTFRC  achieves  n  flow  shares  by  opening  one  connection  and  sending  at  n  times 
the  sending  rate  of  one  TFRC  connection.  In  situations  where  the  throughput  of  each  TFRC 
connection  is  a  function  of  the  random  packet  loss  rate,  e.g.  that  caused  by  random  wireless 
channel  error,  it  is  well  known  the  latter  method  results  in  a  higher  variance  in  the  aggregate 
sending  rate.  Since  the  queueing  delay  along  the  route  is  a  function  of  the  aggregate  sending 
rate,  E-AIOTFRC  experiences  higher  queuing  delay  variance  along  the  path.  Therefore, 
in  order  to  achieve  the  same  level  of  confidence  in  the  measured  queueing  delay,  used  to 
adjust  n  in  Equation  (4.4),  E-AIOTFRC  needs  a  larger  threshold  than  E-MULTFRC.  In 
our  current  implementation,  we  have  empirically  chosen  7  =  0.5  for  E-AIOTFRC. 
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Chapter  5 


Simulations  and  Experiments 


In  this  chapter,  we  carry  out  NS-2  [3]  simulations  and  experiments  over  Verizon  Wireless 
lxRTT  and  EVDO  CDMA  data  network  to  evaluate  the  performance  of  E-MULTCP,  E- 
MULTFRC  and  E-AIOTFRC.  We  use  TCP  NewReno  implementation  for  TCP  protocol  in 
all  simulations. 


5.1  E-MULTCP:  NS-2  Simulations,  lxRTT  and  EVDO 
Wireless  Experiments 

In  this  section,  we  carry  out  NS-2 [3]  simulations  and  actual  experiments  over  Ver¬ 
izon  Wireless  lxRTT  and  EVDO  CDMA  data  network  to  evaluate  the  performance  of 
E-MULTCP. 

5.1.1  Setup 

The  topology  used  in  simulations  is  shown  in  Figure  5.1.  The  sender  denoted  by  s, 
and  the  receiver  denoted  by  r,  both  run  E-MULTCP  at  the  application  layer.  The  number 
of  connections,  as  mentioned  in  the  previous  section,  is  controlled  by  the  sender.  For 
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wireless  link 


Figure  5.1.  Simulation  topology. 

all  simulations,  the  wireless  bandwidth  Bw  is  set  be  1  Mbps,  and  is  assumed  to  be  the 
bottleneck.  The  wireless  link  is  modeled  by  an  exponential  error  model,  and  pw  varies 
from  0.0  to  0.08  in  increments  of  0.02.  DropTail  type  queue  is  used  for  each  node.  In 
order  to  evaluate  E-MULTCP’s  performance  in  the  presence  of  wireless  channel  errors, 
we  examine  three  issues;  first,  how  E-MULTCP  performs  in  terms  of  average  throughput, 
average  round  trip  time,  and  packet  loss  rate,  as  a  function  of  pw.  Second,  whether  the 
number  of  connections  is  stable.  Third,  whether  or  not  an  E-MULTCP  application  can  fairly 
share  with  an  application  using  one  TFRC  or  one  TCP  connection.  In  all  the  simulations, 
throughput  is  measured  every  second,  packet  loss  rate  is  measured  every  30  seconds,  the 
average  round  trip  time  is  measured  every  100  packets,  and  the  number  of  connections  is 
sampled  whenever  there  is  a  change. 

For  the  actual  experiments  over  lxRTT,  we  stream  from  a  desktop  connected  to  Internet 
via  100  Mbps  Ethernet  in  eecs.berkeley.edu  domain,  to  a  notebook  connected  to  Internet 
via  Verizon  Wireless  lxRTT  and  EVDO  CDMA  data  network.  Thus  it  is  quite  likely  that 
the  last  lxRTT  or  EVDO  CDMA  link  is  the  bottleneck  for  the  connection.  We  measure 
the  average  throughput,  average  number  of  connections,  and  packet  loss  rate. 


5.1.2  Performance  Characterization  of  E-MULTCP 

Following  the  analysis  and  arguments  in  Section  4.2.3,  we  use  following  parameters  for  E- 
MULTCP  in  simulations  and  experiments  to  achieve  reasonable  balance  between  utilization 
and  adaptation  rate:  a  =  0.25,  (5  =  0.75.  We  select  7  =  0.2,  and  r  =  20  seconds.  Intuitively, 
larger  r  results  in  more  reliable  estimates  of  round  trip  time,  but  at  the  expense  of  a  lower 
sampling  frequency,  resulting  in  a  less  responsive  system.  Larger  7  results  in  a  system  that 
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is  more  robust  to  round  trip  time  estimates,  but  at  the  expense  of  longer  queues  in  routers, 
and  hence  a  longer  queueing  delay. 

We  simulate  the  E-MULTCP  system  to  send  data  for  9000  seconds,  compute  the  average 
throughput  and  packet  loss  rate  for  pw  =0.0,  0.02,  0.04,  0.06  and  0.081,  and  compare  them 
to  the  optimal,  i.e.  Bw(l  —  pw)  for  each  pw.  The  results  for  Bw  =  1  Mbps ,  (3  =  0.75,  and 
RTTmin  =  168  ms  are  shown  in  Figure  5.2.  As  seen,  the  throughput  is  within  25%  of  the 
optimal,  and  is  reasonably  close  to  the  predicted  one,  i.e.  the  product  of  the  optimal  and 
the  utilization  ratio  computed  using  Equation  (4.3).  The  average  round  trip  time  is  within 
20%  of  RTTmin,  the  same  as  the  expected  range,  i.e.  ^/RTTmin.  and  the  packet  loss  rate 
is  almost  identical  to  the  optimal,  i.e.  a  line  of  slope  one  as  a  function  of  wireless  channel 
error  rate.  As  expected,  the  average  number  of  connections  increases  with  wireless  channel 
error  rate,  pw.  2 

To  demonstrate  the  trade-off  between  utilization  ratio  and  normalized  increasing  rate 
of  E-MULTCP  with  different  values  of  (3,  we  have  also  carried  out  simulations  using  the 
same  setting  as  above,  but  with  (3  =  0.5.  The  results  are  shown  in  Figure  5.3.  As  seen,  the 
throughput  is  lower  than  the  above  simulation  with  (3  =  0.75,  as  expected;  this  is  because 
on  average  fewer  connections  are  opened.  We  will  see  later  the  setting  (3  =  0.5  leads  to  a 
fast  adaptation  to  changes  of  wireless  channel  conditions. 

Considering  the  throughput  plot  in  Figure  5.2,  for  some  values  of  pw,  there  is  a  significant 
difference  between  the  actual  and  optimal  throughput.  This  is  due  to  the  quantization  effect 
in  situations  where  the  number  of  connections  is  small,  i.e.  2  to  4.  In  these  situations,  a 
small  oscillation  around  the  optimal  number  of  connections  results  in  large  variation  in 
observed  throughput.  One  way  to  alleviate  this  problem  is  to  increase  7  in  order  to  tolerate 
larger  queuing  delay  and  hence  absorb  throughput  fluctuations,  at  the  expense  of  being 
less  responsive.  Another  alternative  is  to  use  smaller  packet  size  in  order  to  reduce  the 

1It  is  well  known  that  TCP’s  degraded  performance  is  dominated  by  timeout  if  wireless  packet  loss  rate 
pw  is  higher  than  0.1.  As  our  model  does  not  capture  the  timeout  effect,  we  are  more  interested  in  cases 
where  pw  <  0.1. 

2 Note  the  round  trip  time  for  pw  =  0  is  not  shown  in  Figure  5.2  because  it  represents  the  case  when  the 
channel  is  error  free.  In  this  case,  E-MULTCP  reduces  to  one  TCP  connection. 
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ramping  up  time  (s) 

predicted  (s) 

ramping  down  time  (s) 

predicted  (s) 

/ 3  =  0.75 

660 

650 

80 

73 

13  =  0.5 

313 

325 

32 

36 

Table  5.1.  Adaptation  rates  of  E-MULTCP  with  different  values  of  (3 


’’quantization  effect”  at  the  expense  of  (a)  larger  overhead  and  hence  lower  transmission 
efficiency,  and  (b)  the  slower  rate  of  convergence  to  the  optimal  number  of  connections. 

One  might  also  notice  that  the  difference  between  E-MULTCP ’s  throughput  and  the 
optimal  becomes  larger  as  pw  increases.  This  is  due  to  increased  timeout  events  as  pw 
increases.  Upon  timeout,  TCP  slow  starts  again,  generating  bursty  traffic  that  results 
in  transient  queuing  delay.  Consequently,  detecting  congestion  based  on  RTT  estimate 
becomes  less  reliable,  decreasing  the  average  throughput.  This  effect  is  more  pronounced 
for  large  pw  and  number  of  connections. 

In  order  to  examine  E-MULTCP ’s  adaptation  rate  to  changes  of  wireless  channel  con¬ 
dition,  we  test  E-MULTCP  with  pw  initially  set  at  0.02,  switched  to  0.06  at  3000*ft  second 
and  switched  back  at  6000t/l  second.  The  throughput,  packet  loss  rate,  round  trip  time  and 
the  number  of  opened  connections  are  shown  in  Figure  5.4  for  (3  =  0.75,  and  in  Fig  5.5  for 
/ 3  =  0.5. 

As  seen,  the  number  of  connections  varies  from  around  2-3  to  around  5  as  pw  switches 
from  0.02  to  0.06.  The  ramp  up  and  ramp  down  times  from  2  to  5,  and  5  to  2  connections 
for  (3  =  0.75  and  f3  =  0.5  are  shown  in  Table  5.1.  As  seen,  the  adaptation  performance  is 
close  to  the  theoretical  predictions,  which  are  based  on  analysis  in  Section  4.2.3.  Combined 
with  the  observations  in  Figures  5.2  and  5.3,  we  can  see  that  smaller  f3  results  in  a  faster 
adaptation  rate  to  changes  in  wireless  channel  conditions,  but  a  lower  utilization  ratio. 
These  observations  are  consistent  with  our  analysis  on  the  trade-off  between  utilization 
ratio  and  adaptation  rate,  in  Section  4.2.3. 
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5.1.3  Experimental  Results  for  E-MULTCP  in  2004  and  2005 


Similar  experiments  are  carried  out  on  Verizon  Wireless  lxRTT  CDMA  data  network. 
The  lxRTT  CDMA  data  network  is  advertised  to  operate  at  data  speeds  of  up  to  144  kbps 
for  one  user.  As  we  explore  the  available  bandwidth  for  one  user  using  UDP  flooding,  we 
find  the  highest  average  available  bandwidth  averaged  over  30  minutes  to  be  between  80 
kbps  to  97  kbps.  In  our  experiments,  we  stream  for  30  minutes  from  a  desktop  on  wired 
network  in  eecs.berkeley.edu  domain  to  a  laptop  connected  via  lxRTT  CDMA  modem  using 
E-MULTCP,  E-MULTFRC  and  TCP. 

The  results  are  shown  in  Table  5.2  for  packet  size  of  1460  bytes.  As  seen,  on  average 
E-MULTCP  opens  up  1.7  connections,  and  results  in  58%  higher  throughput,  at  the  expense 
of  a  larger  round  trip  time,  and  higher  packet  loss  rate. 


Table  5.2.  Experimental  results  for  E-MULTCP  and  E-MULTFRC  systems  over  lxRTT 
CDMA. 


scheme 

throughput 

(kbps) 

rtt 

(ms) 

ave.  # 
of  conn. 

throughput  improvement 
over  one  TCP  (%) 

one  TCP 

59 

1954 

N/A 

N/A 

E-MULTCP 

93 

2447 

1.7 

58 

E-MULTFRC 

89 

2767 

1.9 

51 

We  have  also  carried  out  experiments  over  Verizon  Wireless  EVDO  data  network,  by 
sending  5  MB  files  from  a  desktop  on  wired  network  in  eecs.berkeley.edu  domain  to  an 
EVDO  cell  phone  using  E-MULTCP  and  TCP.  A  file  size  of  5  MB  is  chosen  to  be  typical 
of  a  MP3  song.  The  results  are  shown  in  Table  5.3  for  packet  size  of  1460  bytes.  As  seen, 
on  average  E-MULTCP  opens  up  1.8  connections,  and  results  in  43%  higher  throughput, 
at  the  expense  of  a  larger  round  trip  time,  and  higher  packet  loss  rate. 


Table  5.3.  Actual  experimental  results  for  E-MULTCP  and  TCP  over  commercial  EVDO 
data  networks. _ 


scheme 

throughput 

rtt 

ave.  # 

throughput  improvement 

(kbps) 

(ms) 

of  conn. 

over  one  TCP  (%)) 

one  TCP 

249 

702 

N/A 

N/A 

E-MULTCP 

355 

946 

1.8 

43 
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5.1.4  Additional  Experiments  for  E-MULTCP  in  2006 


All  of  the  experiments  in  Section  5.1.3  were  carried  out  in  years  2004  and  2005.  In  Fall 
2005,  Verizon  Wireless  in  USA  launched  a  new  data  service  EV-DO,  which  provides  up  to 
2  Mbps  data  rate.  We  speculate  that  many  lxRTT  users  switched  from  lxRTT  to  EV-DO 
network  due  to  the  higher  data  rates.  Consequently,  as  subscription  level  to  lxRTT  net¬ 
work  dropped,  it  improved  the  overall  throughput  of  the  lxRTT  network  for  the  remaining 
subscribers.  We  repeated  experiments  to  evaluate  E-MULTCP’s  performance  under  new 
channel  conditions  in  Fall  2006  to  reconfirm  improved  performance  of  E-MULTCP  over 
TCP.  The  results  are  reported  in  this  subsection. 

We  implement  E-MULTCP  on  a  laptop  running  Windows  XP,  as  well  as  on  a  cell  phone 
running  BREW  [1].  BREW,  standing  for  Binary  Runtime  Environment  for  Wireless,  is  a 
development  platform  and  coding  language  created  by  Qualcomm  for  cell  phones.  BREW  is 
a  middleware  that  allows  developers  to  easily  port  their  applications  onto  every  cell  phone 
using  Qualcomm  chipsets. 

As  we  explore  the  available  bandwidth  for  one  user  using  UDP  flooding,  we  find  the 
highest  average  available  bandwidth  averaged  over  30  minutes  to  be  between  101  kbps  and 
112  kbps.  This  is  considerably  larger  than  what  we  observed  in  Fall  2004  in  US.  In  the 
experiments  in  2006,  we  send  data  for  1,  2,  5,  and  30  minutes  from  a  desktop  on  wired 
network  in  eecs.berkeley.edu  domain  to  a  laptop  and  to  a  BREW  cell  phone  connected 
via  lxRTT  CDMA  modem  using  E-MULTCP  and  TCP.  The  results  are  shown  in  Table 
5.4  for  packet  size  of  1460  bytes.  Each  data  point  is  averaged  over  5  independent  runs. 
As  seen,  on  average  E-MULTCP  opens  up  around  1.4  connections,  and  results  in  30% 
higher  throughput,  at  the  expense  of  a  larger  round  trip  time,  and  higher  packet  loss  rate. 
Although  these  experiments  result  in  smaller  improvement  over  TCP  than  the  one  shown  in 
Section  5.1.2,  they  still  show  that  E-MULTCP  outperforms  TCP.  The  reason  for  the  smaller 
improvement  is  the  lxRTT  channel  condition  in  2006  being  better  than  in  2004  and  2005. 
From  analysis  in  Section  3.2  and  Equation  3.8,  it  can  be  shown  if  aggregate  packet  loss 
rate  caused  by  channel  error  for  user  r,  i.e.  Y2jerej>  becomes  smaller,  then  utilization  of 
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TCP/TFRC  becomes  higher.  As  TCP/TFRC  achieves  higher  utilization  in  lxRTT  in  2006 
as  compared  to  2004,  there  is  less  room  for  improvement  for  E-MULTCP.  Nevertheless, 
the  insights  still  hold:  as  long  as  TCP/TFRC  performs  in  channel-error-limited  region, 
proposed  solutions  such  as  E-MULTCP  always  improve  throughput. 


Table  5.4.  Actual  experimental  results  for  E-MULTCP  over  lxRTT  network  in  year  2006. 


scheme  and  setup 

throughput 

(kbps) 

rtt 

(ms) 

ave.  # 
of  conn. 

one  TCP  (1  min) 

81 

712 

N/A 

E-MULTCP  (1  min) 

107 

1130 

1.3 

Improvement  (%) 

32 

N/A 

N/A 

one  TCP  (2  min) 

84 

708 

N/A 

E-MULTCP  (2  min) 

110 

1340 

1.3 

Improvement  (%) 

31 

N/A 

N/A 

one  TCP  (5  min) 

86 

723 

N/A 

E-MULTCP  (5  min) 

108 

1372 

1.3 

Improvement  (%) 

25 

N/A 

N/A 

one  TCP  (30  min) 

85 

714 

N/A 

E-MULTCP  (30  min) 

108 

1436 

1.3 

Improvement  (%) 

27 

N/A 

N/A 

5.1.5  Fairness  between  E-MULTCP  and  TCP 

To  investigate  the  fairness  of  E-MULTCP,  we  carry  out  NS-2  simulations  based  on  the 
“dumbbell”  topology  shown  in  Figure  5.6.  Senders  are  denoted  by  si,i  =  1, . . . ,  16,  and 
receivers  are  denoted  by  di,i  =  1 . . . ,  16.  We  investigate  two  types  of  fairness:  the  inter¬ 
protocol  fairness  between  E-MULTCP  and  TCP,  and  the  intra-protocol  fairness  within 
E-MULTCP. 

The  intra-protocol  fairness  is  defined  as  the  fairness  between  E-MULTCP  flows.  In  our 
simulations,  we  run  E-MULTCP  on  all  16  sender-receiver  pairs  shown  in  Figure  5.6  for  5000 
seconds,  and  compare  their  throughput.  E-MULTCP  is  said  to  be  intra-protocol  fair  if  all 
receivers  achieve  the  same  throughput.  The  fairness  ratios  for  pw  =  0.01  and  pw  =  0.04  are 
shown  in  Table  5.5.  The  fairness  ratio  is  defined  as  receivers’  throughput  divided  by  the 
average  throughput;  the  closer  to  one,  the  more  fair  the  E-MULTCP  system  is.  As  seen, 
the  fairness  ratio  is  fairly  close  to  one,  indicating  E-MULTCP  flows  are  fair  to  each  other, 
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at  least  in  this  simulation  setting.  The  bandwidth  utilization  ratios  are  96%  for  pw  =  0.01 
and  98%  for  pw  =  0.04. 


Table  5.5.  Simulation  results  for  intra-protocol  fairness  of  E-MULTCP. 


receiver 

fairness 

ratio 

pu,=0.01 

fairness 

ratio 

pw=0M 

receiver 

fairness 

ratio 

p™=0.01 

fairness 

ratio 

p„,=0.04 

dl 

1.06 

1.03 

d9 

1.07 

0.98 

d2 

1.00 

1.01 

dlO 

0.97 

0.98 

d3 

1.06 

1.03 

dll 

1.02 

1.02 

d4 

1.02 

1.08 

dl2 

1.02 

0.99 

d5 

0.90 

0.98 

dl3 

0.98 

0.97 

d6 

0.93 

0.98 

dl4 

0.89 

1.00 

d7 

1.02 

1.04 

dl5 

1.01 

1.01 

d8 

0.99 

1.01 

dl6 

0.98 

0.94 

The  inter-protocol  fairness  is  defined  as  the  fairness  between  E-MULTCP  and  TCP.  In 
our  simulations,  we  run  E-MULTCP  on  the  first  8  sender-receiver  pairs,  i.e.  ( si,di),i  = 

1, _ ,  8,  and  TCP  on  the  remaining  8  sender-receiver  pairs  shown  in  Figure  5.6;  each  session 

lasts  5000  seconds,  and  we  compare  their  throughput  for  pw  =  0.01  and  pw  =  0.02.  Under 
the  simulation  settings,  each  E-MULTCP  consumes  more  bandwidth  than  one  TCP  under 
full  utilization.  This  is  because  the  wireless  channel  error  rate  is  large  enough  to  make 
the  number  of  virtual  connections  of  each  E-MULTCP  to  be  larger  than  one.  Hence, 
it  is  meaningless  to  define  the  fairness  between  E-MULTCP  and  TCP  as  having  the  same 
throughput.3  As  such,  in  our  simulations,  we  define  E-MULTCP  to  be  fair  to  TCP  if  it  does 
not  result  in  a  significant  decrease  in  TCP’s  throughput  as  compared  to  TCP  throughput  in 
the  absence  of  E-MULTCP.  Specifically  for  our  simulations,  it  implies  TCP  retains  the  same 
throughput  whether  or  not  it  coexists  with  E-MULTCP  under  the  same  network  setting. 
The  throughput  of  E-MULTCP  and  TCP,  as  well  as  the  total  bandwidth  utilization  ratios 
for  the  setup  shown  in  Figure  5.6,  are  shown  in  Table  5.6  for  two  scenarios:  (a)  8  E- 
MULTCP  coexisting  with  8  TCP  connections,  (b)  16  TCP  connections.  Figures  5.7  and  5.8 
also  show  the  dynamics  of  throughput,  packet  loss  rate,  RTT,  and  the  number  of  virtual 

3  Obviously,  there  are  situations  in  which  E-MULTCP  ends  up  with  performing  similar  to  one  TCP. 
An  example  would  be  E-MULTCP  competing  for  bandwidth  with  TCP  on  wired  networks.  In  that  case, 
however,  the  fairness  between  E-MULTCP  and  TCP  is  reduced  to  the  fairness  between  TCP  connections 
themselves,  and  has  been  well  explored  in  literature. 


72 


connections  n  for  the  case  where  8  TCP  connections  coexist  with  8  E-MULTCP  connections. 


Comparing  E-MULTCP+TCP  with  TCP-alone  in  Table  5.6,  we  see  the  former  achieves 
a  much  higher  utilization  of  the  wireless  bandwidth  at  the  expense  of  slightly  lower  TCP 
throughput.  A  careful  examination  of  Figure  5.7  reveals  that  this  throughput  drop  is  caused 
by  the  higher  RTT  experienced  by  TCP  connections  in  the  E-MULTCP+TCP  scenario  as 
compared  with  TCP-alone  scenario.  For  example,  for  pw  =  0.01  and  7  =  0.2,  The  RTT 
for  E-MULTCP+TCP  is  around  0.56  seconds,  while  for  TCP-alone  is  0.5  seconds,  i.e.  the 
propagation  delay4.  As  TCP’s  throughput  is  known  to  be  inversely  proportional  to  RTT, 
the  12%  increase  in  the  RTT  of  TCP  connections  roughly  explains  the  11%  decrease  in  the 
TCP’s  throughput  shown  in  first  row  in  Table  5.6. 


Table  5.6.  Simulation  results  for  fairness  between  E-MULTCP  and  TCP. 


settings 

8  E-MULTCP  +  8  TCP 

16  TCP 

ave.  thput. 
(E-MULTCP) 
(kbps) 

ave.  thput. 
(TCP) 
(kbps) 

utili¬ 

zation 

(%) 

ave.  thput. 
(TCP) 
(kbps) 

utili¬ 

zation 

(%) 

pw=0.01 

7=0.2 

432.1 

160.3 

96 

179.8 

58 

pw=0.01 

7=0.1 

402.3 

170.0 

93 

179.8 

58 

pw= 0.02 
7=0.2 

468.8 

107.8 

94 

120.4 

40 

£+=0.02 

7=0.1 

446.1 

114.3 

92 

120.4 

40 

This  increase  in  the  RTT  experienced  by  TCP  is,  by  design,  a  consequence  of  E- 
MULTCP  controlling  n  according  to  Equation  (4.4).  As  n  is  only  decreased  after  the  queuing 
delay  exceeds  the  threshold  'yrttjmin ,  round  trip  time  is  increased  when  E-MULTCP  in¬ 
creases  n  to  achieve  full  utilization.  One  way  to  address  this  problem  is  to  use  a  smaller 
value  for  7,  in  order  to  reduce  the  increase  in  the  RTT,  and  hence  minimize  the  TCP’s 
throughput  drop.  However,  smaller  values  of  7  also  result  in  lower  bandwidth  utilization 
due  to  increased  sensitivity  of  E-MULTCP  to  fluctuations  in  RTT  measurement.  As  shown 
in  Table  5.6,  the  drop  in  TCP  throughput  for  7  =  0.1  is  smaller  than  that  of  7  =  0.2.  Also, 

4In  this  NS  simulation,  TCP  and  E-MULTCP  share  the  same  route  and  hence  both  have  the  same  round 
trip  time. 
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7  =  0.1  results  in  a  lower  utilization  than  7  =  0.2.  Regardless,  the  stability  of  the  network 
with  mixed  TCP  and  E-MULTCP  is  guaranteed  by  Theorem  3.3.3. 

5.1.6  Slow  Start 

In  practice,  the  optimal  number  of  connections  might  be  a  large  number.  For  instance,  in 
a  Gigabit  network  with  undesirable  hardware  glitches  causing  packets  to  drop  unnecessarily, 
sometimes  more  than  20  connections  are  needed  in  order  to  achieve  full  utilization  of  Gigabit 
links. 

However,  since  E-MULTCP  applies  inversely  increase  law  on  the  number  of  connections 
upon  no  congestion,  it  takes  a  long  time  for  E-MULTCP  to  increase  the  number  of  con¬ 
nections  to  the  optimal,  in  the  initial  starting  stage.  For  instance,  if  the  optimal  number 
of  connections  is  n*  =  20,  using  the  analysis  in  Section  4.2.3,  it  takes  E-MULTCP  about 
16,000  seconds  to  open  20  connections,  which  is  about  4.5  hours.  This  is  too  conservative. 

In  order  to  speed  up  the  initial  climbing,  we  offer  an  optional  technique  that  is  imple¬ 
mented  in  E-MULTCP,  namely  slow  start.  This  is  named  after  the  famous  TCP  slow  start 
technique,  and  in  fact  uses  the  same  idea  as  TCP’s  slow  start.  With  this  technique,  in  the 
initial  climbing  stage,  E-MULTCP  doubles  the  number  of  connections  each  20  seconds  if  no 
congestion  is  observed,  and  halves  as  it  otherwise,  as  follows: 

l/2n  +  a/n,  if  ave_rtt  —  rtt_min  >  'yrtt.min ; 
n  =  {  (5.1) 

2  n,  otherwise. 

By  using  slow  start  technique,  the  number  of  connections  n  in  fact  increases  exponentially 
in  its  increasing  stage.  To  reach  n*  takes  rlog2n*  seconds,  instead  of  ^[( n *)2  —  1]  seconds 
in  normal  operation  stage. 

The  initial  climbing  stage  ends  when  the  first  sign  of  congestion  is  observed,  and  subse¬ 
quently  E-MULTCP  uses  the  HMD  control  law  on  the  number  of  connections.  We  have  car¬ 
ried  out  a  NS-2  simulations  showing  performance  of  slow  start  implemented  in  E-MULTCP. 
The  simulation  topology  is  the  same  as  the  one  shown  in  Figure  5.1,  with  pw  =  0.05, 
Bw  =  10 M  bps,  r  =  10  seconds,  and  propagation  delay  80  ms.  Consequently,  n*  =  24.5. 
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The  simulation  results  are  shown  in  Figure  5.9.  As  seen,  after  applying  slow  start,  the 


initial  climb  up  time  is  about  50  seconds,  which  is  close  to  expected  values  rlog2n*.  This 
is  much  shorter  than  using  HMD  in  the  initial  climbing  stage,  which  is  11,500  seconds. 
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Figure  5.2.  NS-2  simulations  of  E-MULTCP  for  Bw  =  1  Mbps  and  RTTmin  =  168  ms,  for 
P  =  0.75;  (a)  throughput,  (b)  end-to-end  packet  loss  rate,  (c)  end-to-end  round  trip  time, 
(d)  number  of  connections,  all  as  a  function  of  packet  level  wireless  channel  error  rate. 
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Figure  5.3.  NS-2  simulations  of  E-MULTCP  for  Bw  =  1  Mbps  and  RTTmin  =  168  ms,  for 
(3  =  0.5;  (a)  throughput,  (b)  end-to-end  packet  loss  rate,  (c)  end-to-end  round  trip  time, 
(d)  number  of  connections,  all  as  a  function  of  packet  level  wireless  channel  error  rate. 
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Figure  5.4.  NS-2  simulation  results  of  E-MULTCP  with  /3  =  0.75  as  pw  changes  from  0.02 
to  0.06  and  back  again,  for  a  =  0.25;  (a)  end-to-end  round  trip  time,  (b)  throughput,  (c) 
number  of  connections,  (d)  end-to-end  packet  loss  rate,  all  as  a  function  of  time. 
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Figure  5.5.  NS-2  simulation  results  of  E-MULTCP  with  (3  =  0.5  as  pw  changes  from  0.02 
to  0.06  and  back  again;  (a)  end-to-end  round  trip  time,  (b)  throughput,  (c)  number  of 
connections,  (d)  end-to-end  packet  loss  rate,  all  as  a  function  of  time. 
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Figure  5.6.  The  simulation  topology  for  E-MULTCP’s  fairness  evaluation. 
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Figure  5.7.  NS-2  simulation  results  for  the  case  pw  =  0.01,7  =  0.2  for  E-MULTCP+TCP 
scenario:  the  dynamics  of  (a)  throughput,  (b)  number  of  connections  ,  (c)  end-to-end  packet 
loss  rate,  (d)  end-to-end  RTT,  all  as  a  function  of  time. 
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Figure  5.8.  NS-2  simulation  results  for  the  case  pw  =  0.01,7  =  0.1  for  E-MULTCP+TCP 
scenario:  (a)  throughput,  (b)  number  of  connections  ,  (c)  end-to-end  packet  loss  rate,  (d) 
end-to-end  RTT,  all  as  a  function  of  time. 
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Figure  5.9.  NS-2  simulation  results  for  the  case  pw  =  0.05  for  E-MULTCP  with  slow  start: 
(a)  throughput,  (b)  number  of  connections  ,  (c)  end-to-end  packet  loss  rate,  (d)  end-to-end 
RTT,  all  as  a  function  of  time. 


83 


5.2  E-MULTFRC:  NS-2  Simulations  and  lxRTT  Wireless 


Experiments 

In  this  section,  we  carry  out  NS-2 [3]  simulations  and  actual  experiments  over  Verizon 
Wireless  lxRTT  CDMA  data  network  to  evaluate  the  performance  of  E-MULTFRC. 

5.2.1  Setup 

The  topology  used  in  E-MULTFRC  simulations  is  the  same  as  E-MULTCP,  as  shown  in 
Figure  5.1.  The  sender  denoted  by  s,  and  the  receiver  denoted  by  r,  both  run  E-MULTFRC 
at  the  application  layer.  Other  simulation  settings  are  the  same  as  those  in  E-MULTCP 
simulations  as  well. 

Similarly,  in  order  to  evaluate  E-MULTFRC ’s  performance  in  the  presence  of  wireless 
channel  errors,  we  examine  the  same  three  issues  as  in  E-MULTCP  simulations;  first,  how 
E-MULTFRC  performs  in  terms  of  average  throughput,  average  round  trip  time,  and  packet 
loss  rate,  as  a  function  of  pw.  Second,  whether  the  number  of  connections  is  stable.  Third, 
whether  or  not  an  E-MULTFRC  application  can  fairly  share  with  an  application  using  one 
TFRC  or  one  TCP  connection. 

For  the  actual  experiments  over  lxRTT,  we  stream  from  a  desktop  connected  to  Internet 
via  100  Mbps  Ethernet  in  eecs.berkeley.edu  domain,  to  a  notebook  connected  to  Internet 
via  Verizon  Wireless  lxRTT  CDMA  data  network.  The  streaming  takes  30  minutes. 

5.2.2  Performance  Characterization  of  E-MULTFRC 

We  use  the  same  set  of  parameters  a,/?, 7,  and  r  as  E-MULTCP:  a  =  0.25,/?  =  0.75, 
7  =  0.2,  and  r  =  20  seconds. 

We  simulate  the  E-MULTFRC  system  to  stream  for  9000  seconds,  and  compute  the 
average  throughput  and  packet  loss  rate  for  pw  =0.0,  0.02,  0.04,  0.06  and  0.08,  and  compare 
them  to  the  optimal,  i.e.  Bw(l  —  pw)  for  each  pw.  The  results  for  Bw  =  1  Mbps  and 
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RTTmin  =  168  ms  are  shown  in  Figure  5.10.  As  seen,  E-MULTFRC  has  similar  performance 
as  compared  to  E-MULTCP.  The  throughput  is  within  25%  of  the  optimal.  The  average 
round  trip  time  is  within  20%  of  RTTmin ,  the  same  as  the  expected  range  i.e.  7 RTTmin. 
The  packet  loss  rate  is  almost  identical  to  the  optimal,  i.e.  a  line  of  slope  one  as  a  function 
of  wireless  channel  error  rate.  As  expected,  the  average  number  of  connections  increases 
with  wireless  channel  error  rate,  pw.  5 

Considering  the  throughput  plot  in  Figure  5.10,  we  notice  that  for  some  values  of  pw , 
there  is  a  significant  difference  between  the  actual  and  optimal  throughput.  This  is  the 
’’quantization  effect”  in  E-MULTCP’s  simulations.  One  way  to  alleviate  this  problem  is  to 
increase  7  in  order  to  tolerate  larger  queuing  delay  and  hence  absorb  throughput  fluctua¬ 
tions,  at  the  expense  of  being  less  responsive.  Another  alternative  is  to  use  smaller  packet 
size  in  order  to  reduce  the  ’’quantization  effect”  at  the  expense  of  (a)  lower  transmission 
efficiency  and  (b)  the  slower  rate  of  convergence  to  the  optimal  number  of  connections. 

To  examine  the  dynamics  of  E-MULTFRC  system,  we  show  throughput,  packet  loss 
rate,  and  the  number  of  connections  as  a  function  of  time  for  pw  =  0.04  in  Figure  5.11.  As 
seen,  the  throughput  and  the  number  of  connections  are  quite  stable;  as  expected,  packet 
loss  rate  is  around  0.04  and  round  trip  time  is  low,  and  is  in  agreement  with  the  results 
corresponding  to  pw  =  0.04  in  Figure  5.10.  Similar  results  are  obtained  for  other  values  of 
Pw 

In  order  to  examine  E-MULTFRC ’s  performance  as  a  function  of  pw,  as  well  as  the 
dynamics  of  E-MULTFRC,  we  use  E-MULTFRC  with  pw  initially  set  at  0.02.  Then  at 
3000ih  second,  pw  is  switched  to  0.06,  and  at  6000t/l  second  switched  back  to  0.02.  Here,  we 
artificially  change  pw  to  see  how  E-MULTFRC  adapts  to  the  change  in  pw.  The  throughput, 
packet  loss  rate,  round  trip  time  and  the  number  of  opened  connections  are  shown  in  Figure 
5.12.  As  seen,  the  number  of  connections  varies  from  around  2-3  to  around  5  as  pw  switches 
from  0.02  to  0.06. 

5Note  the  round  trip  time  for  pw  =  0  is  not  shown  in  Figure  5.10  because  it  represents  the  channel  error 
free  case  in  which  E-MULTFRC  reduces  to  one  TFRC  connection. 
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5.2.3  Experimental  Results  for  E-MULTFRC 


Similar  experiments  are  carried  out  on  Verizon  Wireless  lxRTT  CDMA  data  network 
in  Spring  of  2005.  At  that  time,  the  lxRTT  CDMA  data  network  is  found  to  have  the 
highest  average  available  bandwidth  averaged  over  30  minutes  to  be  between  80  kbps  to  97 
kbps,  by  using  UDP  flooding.  In  our  experiments,  we  stream  for  30  minutes  from  a  desktop 
on  wired  network  in  eecs.berkeley.edu  domain  to  a  laptop  connected  via  lxRTT  CDMA 
modem  using  E-MULTFRC  and  TFRC.  The  results  are  shown  in  Table  5.7  for  packet  size 
of  1460  bytes.  As  seen,  on  average  E-MULTFRC  opens  up  1.9  connections,  and  results 
in  65%  higher  throughput,  at  the  expense  of  a  larger  round  trip  time,  and  higher  packet 
loss  rate.  Compared  to  E-MULTCP’s  results  in  Table  5.2,  we  can  see  they  have  similar 
performance. 

Table  5.8  shows  packet  loss  details  of  E-MULTFRC  for  one  of  the  30  minutes  long 
experiments  over  lxRTT.  As  expected,  both  the  packet  loss  rate  and  burstiness  of  the  loss 
increase  as  the  number  of  connections  increases. 


Table  5.7.  Actual  experimental 


results  for  an  E-MULTFRC  system  over  lxRTT  CDMA. 


scheme 

throughput 

rtt 

packet  loss 

ave.  # 

throughput  improvement 

(kbps) 

(ms) 

rate 

of  conn. 

over  one  TFRC  (%) 

one  TFRC 

54 

1624 

0.031 

N/A 

N/A 

E-MULTFRC 

89 

2767 

0.041 

1.9 

65 

1 

lable  5.8. 

Packet  loss  details  of  E-MULTFRC 

#  of  conn. 

%  of 

pkt  loss 

avg.  burst 

std. 

max.  burst 

opened 

time 

rate 

error  length 

dev. 

length 

one 

24.1 

0.016 

2.78 

3.26 

7 

two 

61.1 

0.045 

2.43 

3.33 

10 

three 

14.8 

0.076 

3.45 

8.93 

11 

5.2.4  Performance  Comparison  of  E-MULTFRC  and  TFRC 

We  also  carry  out  simulations  for  the  topology  shown  in  Figure  5.13  with  the  following 
settings:  C\  =  1  Mbps ,  C2  =  5  Mbps,  S  =  760  bytes,  users  1,  2,  3  have  round  trip  propaga¬ 
tion  delays  of  691  ms,  651  ms,  and  69  ms  respectively.  We  use  two  sets  of  p,/,).  The  first 
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set  is  (0,0),  representing  the  wired  scenario;  the  second  set  is  (0.02,0.01),  representing  the 
wireless  scenario.  The  wireless  link  is  modeled  as  a  wired  link  with  an  exponential  random 
packet  loss  model. 

In  simulations,  we  stream  9000  seconds  video  from  Si  to  r*  for  user  i,  i  =  1,2,3,  using 
E-MULTFRC  scheme,  or  only  one  TFRC.  The  simulation  results  for  TFRC  are  shown  in 
Figure  5.14  for  wireless  scenario  with  =  0.02  and  py,  =  0.01.  Those  for  E-MULTFRC  are 
shown  in  Figure  5.15  for  wireless  scenario  with  =  0.02  and  py,  =  0.01,  and  in  Figure  5.16 
wired  scenario  with  p^  =  p^  =  0.  Based  on  this  setting,  the  minimum  end-to-end  packet 
loss  rates  for  user  2  and  3  are  Py,  and  p'y,  respectively,  i.e.  0.02  and  0.01  respectively  in 
the  wireless  scenario,  and  simply  0  in  the  wired  scenario.  The  minimum  end-to-end  packet 
loss  rate  for  user  1  can  be  easily  computed  as  1  —  (1  —  p^)(l  —  Py,),  which  is  0.0297  for  the 
wireless  scenario,  and  0  in  the  wired  scenario.  The  minimum  round  trip  times  for  users  1, 
2,  and  3  in  the  wireless  scenario  are  merely  the  propagation  delays,  i.e.  691  ms,  651  ms, 
and  69  ms  respectively. 

The  results  in  Figures  5.14,  5.15  and  5.16  include  throughput  measured  every  10  seconds, 
number  of  connections,  end-to-end  packet  loss  rate  measured  every  30  seconds  and  average 
rtt  measured  every  20  seconds.  Several  observations  can  be  drawn  from  the  Figures.  First, 
as  predicted  from  the  analysis  in  Section  3.3,  when  plw  =  p^  =  0,  as  shown  in  Figure  5.16,  E- 
MULTFRC  opens  one  connection,  thereby  being  reduced  to  one  TFRC  in  the  wired  network 
case.  Second,  comparing  Figures  5.15(a)  and  5.16(a),  E-MULTFRC  can  achieve  almost  the 
same  stationary  sending  rate  in  wireless  and  wired  networks.  Moreover,  as  seen  from  Figure 
5.15(c),  the  end-to-end  packet  loss  rates  for  users  1,  2,  and  3  are  about  0.03,  0.02,  and  0.01, 

1. e.  the  minimum  values  for  the  wireless  scenario,  as  discussed  earlier.  In  addition,  as  shown 
in  Figure  5.16(c),  the  packet  loss  rate  for  all  3  users  in  the  wired  scenario  is  zero,  which  is 
in  agreement  with  the  minimum  values  discussed  earlier.  Since  at  time  zero,  initially  two 
connections  are  opened  before  the  number  of  connections  eventually  converges  to  one  in 
Figure  5.16(c),  there  is  a  spike  in  end-to-end  packet  loss  rate  at  time  zero,  which  eventually 
converges  to  zero.  Similarly,  as  seen  from  Figure  5.15(d),  the  round  trip  times  for  users  1, 

2,  and  3  in  the  wireless  scenario  are  around  the  minimum  values,  as  discussed  above.  These 
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are  simply  the  results  of  E-MULTFRC  pursuing  the  boundary  between  full  utilization  and 
underutilization.  Fourth,  comparing  Figures  5.14(a)  and  5.15(a),  E-MULTFRC  achieves  a 
higher  throughput  than  one  TFRC  scheme  in  wireless  networks,  at  the  expense  of  minor 
modification  to  the  application  layer.  This  confirms  our  analysis  in  Section  3.3,  and  shows 
E-MULTFRC  can  achieve  good  performance  in  the  wireless  scenario. 

5.2.5  Fairness  between  E-MULTFRC  and  TCP 

To  investigate  the  fairness  of  E-MULTFRC,  we  carry  out  NS-2  simulations  based  on 
the  “dumbbell”  topology  shown  in  Figure  5.6.  Senders  are  denoted  by  si,i  =  1, . . . ,  16,  and 
receivers  are  denoted  by  di,i  =  1 . . . ,  16.  We  investigate  two  types  of  fairness:  the  inter¬ 
protocol  fairness  between  E-MULTFRC  and  TCP,  and  the  intra-protocol  fairness  within 
E-MULTFRC. 

The  intra-protocol  fairness  is  defined  as  the  fairness  between  E-MULTFRC  flows.  In 
our  simulations,  we  run  E-MULTFRC  on  all  16  sender-receiver  pairs  shown  in  Figure  5.6  for 
5000  seconds,  and  compare  their  throughput.  E-MULTFRC  is  said  to  be  intra-protocol  fair 
if  all  receivers  achieve  the  same  throughput.  The  fairness  ratios  for  pw  =  0.01  and  pw  =  0.04 
are  shown  in  Table  5.9.  The  fairness  ratio  is  defined  as  receivers’  throughput  divided  by  the 
average  throughput;  the  closer  to  one,  the  more  fair  the  E-MULTFRC  system  is.  As  seen, 
the  fairness  ratio  is  fairly  close  to  one,  indicating  E-MULTFRC  flows  are  fair  to  each  other, 
at  least  in  this  simulation  setting.  The  bandwidth  utilization  ratios  are  96%  for  pw  =  0.01 
and  98%  for  pw  =  0.04. 

The  inter-protocol  fairness  is  defined  as  the  fairness  between  E-MULTFRC  and  TCP. 
In  our  simulations,  we  run  E-MULTFRC  on  the  first  8  sender-receiver  pairs,  i.e.  (si,  di),i  = 
1, . . . ,  8,  and  TCP  on  the  remaining  8  sender-receiver  pairs  shown  in  Figure  5.6;  each  session 
lasts  5000  seconds,  and  we  compare  their  throughput  for  p.w  =  0.01  and  pw  =  0.02.  Under 
the  simulation  settings,  each  E-MULTFRC  consumes  more  bandwidth  than  one  TCP  under 
full  utilization.  This  is  because  in  this  case,  the  wireless  channel  error  rate  is  large  enough 
to  make  the  number  of  virtual  connections  of  each  E-MULTFRC  to  be  larger  than  one. 
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Table  5.9.  Simulation  results  for  intra-protocol  fairness  of  E-MULTFRC. 


receiver 

fairness 

ratio 

pu,=0.01 

fairness 

ratio 

jT,,=0.04 

receiver 

fairness 

ratio 

p™=0.01 

fairness 

ratio 

7^ -0.04 

dl 

1.08 

1.03 

d9 

1.04 

0.98 

d2 

0.99 

1.01 

dlO 

0.98 

0.98 

d3 

1.05 

1.03 

dll 

1.03 

1.02 

d4 

1.01 

1.08 

dl2 

1.03 

0.99 

d5 

0.91 

0.98 

dl3 

1.00 

0.97 

d6 

0.92 

0.98 

dl4 

0.90 

1.00 

d7 

1.02 

1.04 

dl5 

1.02 

1.01 

d8 

0.98 

1.01 

dl6 

1.00 

0.94 

Hence,  it  is  meaningless  to  define  the  fairness  between  E-MULTFRC  and  TCP  as  having 
the  same  throughput.6  As  such,  in  our  simulations,  we  define  E-MULTFRC  to  be  fair  to 
TCP  if  it  does  not  result  in  a  decrease  in  TCP’s  throughput  as  compared  to  the  case  where 
E-MULTFRC  is  absent.  Specifically  for  our  simulations,  it  implies  TCP  retains  the  same 
throughput  whether  or  not  it  coexists  with  E-MULTFRC  under  the  same  network  setting. 
The  throughput  of  E-MULTFRC  and  TCP,  as  well  as  the  total  bandwidth  utilization  ratios 
for  the  setup  shown  in  Figure  5.6,  are  shown  in  Table  5.10  for  two  scenarios:  (a)  8  E- 
MULTFRC  coexisting  with  8  TCP  connections,  (b)  16  TCP  connections.  Figures  5.17 
and  5.18  also  show  the  dynamics  of  throughput,  packet  loss  rate,  RTT,  and  the  number  of 
virtual  connections  n  for  7  =  0.2  and  7  =  0.1  respectively.  Comparing  E-MULTFRC+TCP 
with  TCP-alone  in  Table  5.10,  we  see  the  former  achieves  a  much  higher  utilization  of  the 
wireless  bandwidth  at  the  expense  of  lower  TCP  throughput.  A  careful  examination  of 
Figure  5.17  reveals  that  this  throughput  drop  is  caused  by  the  higher  RTT  experienced 
by  TCP  connections  in  the  E-MULTFRC+TCP  scenario  as  compared  with  TCP-alone 
scenario.  For  example,  for  pw  =  0.01  and  7  =  0.2,  E-MULTFRC+TCP  experiences  around 
0.55  seconds  RTT,  while  TCP-alone  only  experiences  0.5  seconds  RTT,  i.e.  the  propagation 
delay7.  As  TCP’s  throughput  is  known  to  be  inversely  proportional  to  RTT,  the  10% 

6Obviously,  there  are  situations  in  which  E-MULTFRC  ends  up  with  performing  similar  to  one  TFRC. 
An  example  would  be  E-MULTFRC  competing  for  bandwidth  with  TCP  on  wired  networks.  In  that  case, 
however,  the  fairness  between  E-MULTFRC  and  TCP  is  reduced  to  the  fairness  between  TFRC  and  TCP, 
and  has  been  well  explored  in  [23]. 

7 In  this  NS  simulation,  TCP  and  E-MULTFRC  share  the  same  route  and  hence  both  have  the  same 
round  trip  time. 
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increase  in  the  RTT  of  TCP  connections  roughly  explains  the  13%  decrease  in  the  TCP’s 
throughput  shown  in  first  row  in  Table  5.10. 


Table  5.10.  Simulation  results  for  fairness  between  E-MULTFRC  and  TCP. 


settings 

8  E-MULTFRC  +  8  TCP 

16  TCP 

ave.  thput. 
(E-MULTFRC) 
(kbps) 

ave.  thput. 
(TCP) 
(kbps) 

utili¬ 

zation 

(%) 

ave.  thput. 
(TCP) 
(kbps) 

utili¬ 

zation 

(%) 

p^=0.01 

7=0.2 

436.59 

176.67 

99 

200.168 

65 

pw=0.01 

7=0.1 

408.84 

188.40 

97 

200.168 

65 

pw= 0.02 
7=0.2 

472.52 

127.81 

98 

139.674 

46 

pw= 0.02 
7=0.1 

444.90 

134.64 

93 

139.674 

46 

This  increase  in  the  RTT  experienced  by  TCP  is,  by  design,  a  consequence  of  E- 
MULTFRC  controlling  n  according  to  Equation  (4.4).  As  n  is  only  decreased  after  the  queu¬ 
ing  delay  exceeds  the  threshold  7 rttjnin,  round  trip  time  is  increased  when  E-MULTFRC 
increases  n  to  achieve  full  utilization.  One  way  to  address  this  problem  is  to  use  a  smaller 
value  for  7,  in  order  to  reduce  the  increase  in  the  RTT,  and  hence  minimize  the  TCP’s 
throughput  drop.  However,  similar  to  the  arguments  in  E-MULTCP’s  simulations  in  Sec¬ 
tion  5.1.3,  for  smaller  values  of  7  also  result  in  lower  bandwidth  utilization  due  to  increased 
sensitivity  of  E-MULTFRC  to  fluctuations  in  measured  RTT.  As  shown  in  Table  5.10, 
7  =  0.1  results  in  a  smaller  negative  change  in  the  TCP’s  throughput,  than  7  =  0.2. 
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Figure  5.10.  NS-2  simulations  of  E-MULTFRC  for  Bw  =  1  Mbps  and  RTTmin  =  168  ms; 
(a)  throughput,  (b)  end-to-end  packet  loss  rate,  (c)  end-to-end  round  trip  time,  (d)  number 
of  connections,  all  as  a  function  of  packet  level  wireless  channel  error  rate. 
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Figure  5.11.  NS-2  simulations  of  E-MULTFRC  for  Bw  =  1  Mbps  and  pw  =  0.04;  (a)  end- 
to-end  round  trip  time,  (b)  throughput,  (c)  end-to-end  packet  loss  rate,  (d)  number  of 
connections,  all  as  a  function  of  time. 
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Figure  5.12.  NS-2  simulation  results  of  E-MULTFRC  as  pw  changes  from  0.02  to  0.06  and 
back  again;  (a)  end-to-end  round  trip  time,  (b)  throughput,  (c)  numbers  of  connections, 
(d)  end-to-end  packet  loss  rate,  all  as  a  function  of  time. 


Figure  5.13.  Simulation  topology. 
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Figure  5.14.  NS-2  simulations  of  one  TFRC  for  the  topology  shown  in  Figure  5.13  with 
plj  =  0.02,  =  0.01:  (a)  throughput,  (b)  number  of  connections,  (c)  end-to-end  packet 

loss  rate  and  (d)  average  rtt. 
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Figure  5.15.  NS-2  simulations  of  E-MULTFRC  for  the  topology  shown  in  Figure  5.13  with 
plj  =  0.02,  =  0.01:  (a)  throughput,  (b)  number  of  connections,  (c)  end-to-end  packet 

loss  rate  and  (d)  average  rtt. 
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Figure  5.16.  NS-2  simulations  of  E-MULTFRC  for  the  topology  shown  in  Figure  5.13  with 
plj  =  0.00,  =  0.00:  (a)  throughput,  (b)  number  of  connections,  (c)  end-to-end  packet 

loss  rate  and  (d)  average  rtt. 
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Figure  5.17.  NS-2  simulation  results  of  E-MULTFRC  for  the  case  pw  =  0.01,7  =  0.2  for 
E-MULTFRC+TCP  scenario:  the  dynamics  of  (a)throughput,  (b)  number  of  connections  , 
(c)  end-to-end  packet  loss  rate,  (d)  end-to-end  RTT,  all  as  a  function  of  time. 
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Figure  5.18.  NS-2  simulation  results  of  E-MULTFRC  for  the  case  pw  =  0.01,7  =  0.1  for 
E-MULTFRC+TCP  scenario:  (a) throughput,  (b)  number  of  connections  ,  (c)  end-to-end 
packet  loss  rate,  (d)  end-to-end  RTT,  all  as  a  function  of  time. 
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5.2.6  Comparison  Between  E-MULTFRC  and  Video  Transport  Protocol 
(VTP) 

VTP  [57]  is  an  experimental  protocol  with  a  new  end-to-end  rate  control  mechanism 
designed  specifically  for  real-time  streaming  in  wireless  networks.  It  relies  on  two  techniques, 
namely  the  Achieved  Rate  (AR)  estimation  and  Loss  Discrimination  Algorithm  (LDA).  AR 
is  the  receiving  rate  measured  by  receiver,  plus  the  fraction  corresponding  to  packet  loss 
caused  by  wireless  error.  VTP  uses  an  end-to-end  statistics  based  LDA  called  Spike  that 
infers  congestion  based  on  the  measured  RTT,  and  switches  between  a  congestion  and  an 
error  state  [57]. 

Similar  to  TCP,  VTP  linearly  probes  the  available  bandwidth  until  congestion  is  de¬ 
tected.  However,  VTP  does  not  perform  multiplicative  decrease  upon  congestion,  instead  it 
reduces  the  sending  rate  to  AR  and  avoids  the  drastic  rate  reductions  for  improved  smooth¬ 
ness,  at  the  same  time  maintaining  the  same  average  rate  as  TCP.  In  contrast  to  the  highly 
fluctuating  TCP  rate,  VTP  reduces  its  rate  by  a  lower  amount  but  keeps  this  reduced  rate 
for  longer.  It  is  easy  to  see  that  in  the  steady  state  in  VTP,  the  average  sending  rate  is 
equal  to  AR. 

VTP  belongs  to  the  end-to-end  statistics  based  solution  for  TFRC  over  wireless,  as 
mentioned  in  Section  1.2.  VTP  is  an  entirely  new  rate  control  protocol  and  hence  requires 
modifications  to  transport  layer  protocol  stack.  Its  effectiveness  has  not  been  evaluated 
either  in  theory  or  over  the  Internet  or  in  practical  wireless  networks. 

In  this  section,  we  briefly  compare  performance  of  E-MULTFRC  and  VTP  using  NS-2 
simulations  over  the  topology  shown  in  5.1.  The  sender  denoted  by  s,  and  the  receiver 
denoted  by  r,  both  run  E-MULTFRC  with  slow  start  option  at  the  application  layer  in  one 
simulation,  and  VTP  at  the  transport  layer  in  the  other  simulation.  A  detailed  comparison 
is  provided  by  Yang  et.  al.  [56]. 

For  all  simulations,  the  wireless  bandwidth  Bw  is  set  be  4  Mbps  and  is  assumed  to  be 
the  bottleneck.  Round  trip  propagation  delay  RTTmin  is  80  ms.  We  send  dummy  data  from 
sender  to  receiver  for  300  seconds,  using  both  E-MULTFRC  and  VTP  schemes.  In  one  set 
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of  comparison,  the  wireless  link  is  modeled  by  an  exponential  error  model,  and  pw  is  set  to 
be  0.02.  In  the  other  set  of  comparison,  the  wireless  link  is  modeled  by  a  two  states  Markov 
model  with  following  settings:  duration  of  good  state  is  1  second;  duration  of  bad  state  is 
0.5  second;  pw  =  0  in  good  state;  pw  =  0.06  in  bad  state;  hence  the  overall  packet  loss  rate 
caused  by  wireless  error  is  2%;  state  transition  matrix  is 

'  0.75  0.25 
^  0.25  0.75 

DropTail  type  queue  is  used  for  each  node.  In  order  to  compare  the  performance  of  E- 
MULTFRC  and  VTP  in  the  presence  of  wireless  channel  errors,  we  examine  their  throughput 
in  the  presence  of  simulated  wireless  random  and  burst  loss.  In  all  the  simulations,  through¬ 
put  is  measured  every  10  seconds,  and  the  number  of  connections  is  sampled  whenever  there 
is  a  change. 

The  throughput  comparison  in  the  presence  of  random  loss  and  burst  loss  are  shown 
in  Figures  5.19  and  5.20,  respectively.  As  seen,  in  the  presence  of  random  wireless  er¬ 
ror,  VTP  has  a  slightly  higher  throughput  than  E-MULTFRC  and  its  utilization  ratio  is 
97%,  compared  to  that  of  E-MULTFRC  at  91  %.  VTP’s  result  is  consistent  with  those 
in  literature:  VTP’s  LDA  works  in  a  scenario  with  one  flow,  one  wireless  link,  and  ran¬ 
dom  loss8.  E-MULTFRC ’s  performance  is  consistent  with  our  analysis  and  simulation  in 
previous  sections. 

On  the  other  hand,  in  the  presence  of  burst  error,  E-MULTFRC  outperforms  VTP  all 
the  time,  and  its  overall  utilization  is  85%,  as  compared  to  VTP’s  70%.  This  is  consistent 
with  the  results  in  [56]  in  that  VTP’s  LDA  works  sub-optinrally  in  burst  loss  scenarios.  E- 
MULTFRC,  on  the  other  hand,  is  not  sensitive  to  burst  errors  happening  in  a  timescale  much 
faster  than  that  of  changing  connections.  Intuitively,  this  is  because  E-MULTFRC  cares 
about  long  term  average  utilization  over  20  seconds,  and  as  such  effects  of  the  short  term 
packet  loss  fluctuation  on  utilization  tend  to  average  out,  and  will  not  affect  E-MULTFRC ’s 
performance  in  general. 

8VTP’s  performance  has  not  been  extensively  evaluated  in  a  simulation  or  real  scenarios  with  multiple 
users  and  various  loss  conditions.  An  extensive  study  of  LDA  algorithms,  including  VTP’s,  is  included  in 

[14]- 
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(b) 


Figure  5.19.  Performance  comparison  between  E-MULTFRC  and  VTP,  in  the  presence  of 
random  loss:  (a)  throughput;  (b)  the  number  of  connections  opened  by  E-MULTFRC. 
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Figure  5.20.  Performance  comparison  between  E-MULTFRC  and  VTP,  in  the  presence  of 
burst  loss:  (a)  throughput;  (b)  the  number  of  connections  opened  by  E-MULTFRC. 
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In  summary,  as  compared  to  VTP,  E-MULTFRC  performs  slightly  worse  in  presence 
of  random  loss,  and  much  better  in  the  presence  of  burst  loss.  Moreover,  E-MULTFRC’s 
performance  has  been  evaluated  in  this  thesis  both  theoretically  and  experimentally  over 
commercial  wireless  networks.  Meanwhile  VTP’s  theoretical  and  experimental  performance 
are  subject  to  further  research.  Last  but  not  the  least,  VTP  is  a  completely  new  rate  con¬ 
trol/congestion  control  protocol,  and  hence  requires  modification  to  transport  layer  protocol 
stack,  i.e.  at  the  operating  system  level,  in  order  to  be  deployed;  E-MULTFRC  requires  no 
modification  to  existing  transport  layer  protocol  stack  and  underlying  infrastructure,  and 
is  potentially  easier  to  deploy. 


5.3  MULTFRC:  NS-2  Simulations  and  lxRTT  Wireless  Ex¬ 
periments 

In  this  section,  we  carry  out  NS-2  simulations  and  actual  experiments  over  Verizon 
Wireless  lxRTT  CDMA  data  network  to  evaluate  the  performance  of  MULTFRC  system. 

5.3.1  Setup 

The  topology  used  in  simulations  is  the  same  as  E-MULTFRC’s  simulations,  as  shown 
in  Figure  5.1.  The  sender  denoted  by  s,  and  the  receiver  denoted  by  r,  both  run  MULTFRC 
at  the  application  layer. 

We  examine  two  issues;  first,  how  MULTFRC  performs  in  terms  of  average  throughput, 
average  round  trip  time,  and  packet  loss  rate,  as  a  function  of  pw.  Second,  whether  the 
number  of  connections  is  stable. 

For  the  actual  experiments  over  lxRTT,  we  stream  from  a  desktop  connected  to  Internet 
via  100  Mbps  Ethernet  in  EECS,  Berkeley,  to  a  notebook  connected  to  Internet  via  Verizon 
Wireless  lxRTT  CDMA  data  network.  The  packet  size  S  is  1460  bytes,  and  the  streaming 
takes  30  minutes. 
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5.3.2  Performance  Characterization  of  MULTFRC 


We  have  empirically  found  the  following  parameters  to  result  in  reasonable  performance: 
a  =  (3  =  1,7  =  0.2  and  m  =  50. 

We  simulate  the  MULTFRC  system  to  stream  for  9000  seconds,  and  compute  the  average 
throughput  and  packet  loss  rate  for  pw  =0.0,  0.02,  0.04,  0.06  and  0.08,  and  compare  them 
to  the  optimal,  i.e.  Bw(  1  —  pw)  for  each  pw.  The  results  for  Bw  =  1  Mbps  and  RTTmin  = 
168  ms  are  shown  in  Figure  5.21.  As  seen,  the  throughput  is  within  25%  of  the  optimal.  The 
round  trip  time  is  within  20%  of  RTTmin ,  the  same  as  the  expected  range,  i.e.  7 RTTm,in. 
The  packet  loss  rate  is  almost  identical  to  the  optimal,  i.e.  a  line  of  slope  one  as  a  function 
of  wireless  channel  error  rate.  As  expected,  the  average  number  of  connections  increases 
with  wireless  channel  error  rate,  pw.  To  confirm  MULTFRC ’s  performance  over  a  wider 
range  of  parameters,  we  carry  out  additional  simulations  using  the  same  topology  as  in 
Figure  5.1,  with  Bw  =  100  kbps  and  RTTmin  =  757  ms.  The  results,  shown  in  Figure  5.22 
are  as  expected,  and  validate  our  earlier  observations.  9 

Considering  the  throughput  plots  in  Figures  5.21  and  5.22,  we  notice  that  for  some  values 
of  pw ,  there  is  a  significant  difference  between  the  actual  and  optimal  throughput.  This  is 
due  to  the  quantization  effect  in  situations  where  the  number  of  connections  is  small,  i.e.  2  to 
4.  In  these  situations,  a  small  oscillation  around  the  optimal  number  of  connections  results 
in  large  variation  in  observed  throughput.  One  way  to  alleviate  this  problem  is  to  increase 
7  in  order  to  tolerate  larger  queuing  delay  and  hence  absorb  throughput  fluctuations,  at  the 
expense  of  being  less  responsive.  Another  alternative  is  to  use  smaller  packet  size  in  order 
to  reduce  the  ’’quantization  effect”  at  the  expense  of  (a)  lower  transmission  efficiency  and 
(b)  the  slower  rate  of  convergence  to  the  optimal  number  of  connections.  As  discussed  in 
Section  4.5,  it  is  also  possible  to  combine  the  multiple  TFRC  connection  into  one,  in  order 
to  mitigate  quantization  effect. 

The  comparison  between  the  performance  of  E-MULTFRC  and  MULTFRC  in  Figure 
5.10  also  shows  the  difference  caused  by  applying  different  control  laws  in  these  two  schemes. 

9Note  the  round  trip  times  for  pw  =  0  are  shown  neither  in  Figure  5.21  or  5.22  because  they  represent 
the  channel  error  free  case  in  which  MULTFRC  reduces  to  one  TFRC  connection. 
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When  pw  is  small,  e.g.  0.02,  the  optimal  number  of  connections  is  small.  In  this  case,  E- 
MULTFRC  outperforms  MULTFRC  since  when  decreasing  the  number  of  connections  n, 
E-MULTFRC  only  decreases  it  proportionally  whereas  MULFRC  decrements  it.  On  the 
other  hand,  when  pw  is  large,  e.g.  0.08,  the  optimal  number  of  connections  is  large.  In  this 
case,  the  proportional  decrease  in  n  is  more  significant  than  decrementing  it.  Therefore,  as 
the  behavior  of  E-MULTFRC  and  MULTFRC  are  similar  when  increasing  the  number  of 
connections,  the  one  with  more  significant  decrease  in  n  will  have  lower  average  throughput. 
Hence,  when  channel  condition  is  worse  and  the  packet  loss  rate  caused  by  channel  error 
is  high,  MULTFRC  will  have  a  better  utilization  than  E-MULTFRC,  due  to  the  advantage 
of  HAD  control  law  on  the  number  of  connections.  This  confirms  the  utilization  ratio 
comparisons  in  Sections  4.2.3  and  4.4  for  E-MULTFRC  and  MULTFRC,  respectively. 

To  examine  the  dynamics  of  MULTFRC  system,  we  show  throughput,  packet  loss  rate, 
and  the  number  of  connections  as  a  function  of  time  for  pw  =  0.04  in  Figure  5.23.  As 
seen,  the  throughput  and  the  number  of  connections  are  quite  stable;  as  expected,  packet 
loss  rate  is  around  0.04  and  round  trip  time  is  low,  and  is  in  agreement  with  the  results 
corresponding  to  pw  =  0.04  in  Figure  5.21.  Similar  results  are  obtained  for  other  values  of 
Pw 

In  order  to  examine  MULTFRC ’s  performance  as  a  function  of  pw,  we  use  MULTFRC 
with  pw  initially  set  at  0.02.  Then  at  3000th  second,  pw  is  switched  to  0.08,  and  at  6000th 
second  switched  back  to  0.02.  Here,  we  artificially  change  pw  to  see  how  MULTFRC  adapts 
to  the  change  in  pw.  The  throughput,  packet  loss  rate,  round  trip  time  and  the  number  of 
connections  opened  are  shown  in  Figure  5.24.  As  seen,  the  number  of  connections  varies 
from  around  3  to  around  7  as  pw  switches  from  0.02  to  0.08. 

Using  NS-2  simulations,  we  have  empirically  found  the  fairness  results  between  MULT¬ 
FRC  and  TCP/TFRC  to  be  similar  to  that  of  E-MULTFRC  and  TCP/TFRC.  This  is 
not  surprising  because  from  Corollaries  3.4.1  and  3.7.1,  it  is  clear  that  both  E-MULTFRC 
and  MULTFRC  use  only  the  leftover  bandwidth  that  TCP/TFRC  can  not  utilize  in  the 
channel-error-limited  region.  Furthermore,  in  the  capacity-limited  region,  we  have  shown 
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that  E-MULTFRC  and  MULTFRC  open  only  one  connection,  and  as  such  share  bandwidth 
fairly  with  TCP/TFRC. 


5.3.3  Experimental  Results  for  MULTFRC 

As  for  actual  experiments,  we  compare  the  performance  of  MULTFRC,  one  TFRC,  and 
E-MULTFRC  in  Table  5.11.  As  seen,  MULTFRC  on  average  opens  up  1.8  connections,  and 
results  in  60%  higher  throughput  than  one  TFRC  connection  system,  at  the  expense  of  a 
larger  round  trip  time,  and  higher  packet  loss  rate.  As  seen,  the  performance  of  MULTFRC 
and  E-MULTFRC  are  pretty  close. 


Table  5.11.  Actual  experimental  results  for  a  MULTFRC  system  over  lxRTT  CDMA. 


scheme 

throughput 

(kbps) 

rtt 

(ms) 

packet  loss 
rate 

ave.  # 
of  conn. 

throughput  improvement 
over  one  TFRC  (%) 

one  TFRC 

54 

1624 

0.031 

N/A 

N/A 

MULTFRC 

86 

2512 

0.045 

1.8 

60 

E-MULTFRC 

89 

2762 

0.41 

1.9 

65 
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Figure  5.21.  NS-2  simulations  of  MULTFRC  for  Bw  =  1  Mbps  and  RTTmin  =  168  ms;  (a) 
throughput,  (b)  end-to-end  packet  loss  rate,  (c)  end-to-end  round  trip  time,  (d)  number  of 
connections,  all  as  a  function  of  packet  level  wireless  channel  error  rate. 
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Figure  5.22.  NS-2  simulations  of  MULTFRC  for  Bw  =  100  kbps  and  RTTmin  =  757  ?ns;  (a) 
throughput,  (b)  end-to-end  packet  loss  rate,  (c)  end-to-end  round  trip  time,  (d)  number  of 
connections,  all  as  a  function  of  packet  level  wireless  channel  error  rate. 
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Figure  5.23.  NS-2  simulations  of  MULTFRC  for  Bw  =  1  Mbps  and  pw  =  0.04;  (a)  end-to-end 
round  trip  time,  (b)  throughput,  (c)  end-to-end  packet  loss  rate,  (d)  number  of  connections, 
all  as  a  function  of  time. 
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Figure  5.24.  NS-2  simulation  results  of  MULTFRC  as  pw  changes  from  0.02  to  0.08  and 
back  again;  (a)  end-to-end  round  trip  time,  (b)  throughput,  (c)  numbers  of  connections, 
(d)  end-to-end  packet  loss  rate,  all  as  a  function  of  time. 
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5.4  E-AIOTFRC:  Simulation  Results 


In  this  section,  we  carry  out  NS-2  simulations  to  evaluate  the  performance  of  E- 
AIOTFRC.  Specifically,  we  examine  three  issues  in  these  simulations:  (a)  how  E-AIOTFRC 
performs  in  terms  of  average  throughput,  average  RTT,  and  packet  loss  rate,  as  a  function 
of  pVJ .  and  how  it  compares  with  E-MULTFRC;  (b)  whether  n  is  stable;  (c)  whether  or 
not  E-AIOTFRC  is  fair  to  TCP.  In  all  the  simulations,  throughput  is  measured  every  10 
seconds,  packet  loss  rate  is  measured  every  30  seconds,  the  average  RTT  is  measured  every 
100  packets,  and  the  number  of  connections  is  sampled  whenever  there  is  a  change. 

The  topology  used  in  simulations  for  utilization  evaluation  is  again  the  same  as 
EMULTCP’s,  shown  in  Figure  5.1.  The  sender  is  denoted  by  s,  and  the  receiver  is  de¬ 
noted  by  r.  They  both  run  E-AIOTFRC  at  the  application  layer. 

We  simulate  the  E-AIOTFRC  system  to  stream  for  9000  seconds.  The  average  through¬ 
put,  end-to-end  packet  loss  rate,  average  RTT,  and  average  n  for  pw  =0.0,  0.02,  0.04,  0.06 
and  0.08  are  shown  in  Figure  5.25,  where  RTTmin  =  168  ms.  As  seen,  the  throughput  is 
within  15%  of  the  optimal,  and  the  packet  loss  rate  is  almost  identical  to  the  optimal,  i.e. 
a  line  of  slope  one  as  a  function  of  wireless  channel  error  rate.  As  expected,  the  average  n 
increases  with  wireless  channel  error  rate,  pw.  10 

The  throughput  of  E-MULTFRC  is  also  shown  in  Figure  5.25(a)  for  comparison.  As 
seen,  E-AIOTFRC  has  almost  the  same  throughput  as  E-MULTFRC  when  pw  is  high, 
while  it  significantly  outperforms  E-MULTFRC  when  pw  is  low.  For  example,  when  pw  = 
0.02,  E-AIOTFRC  achieves  95%  utilization  of  the  wireless  bandwidth,  while  E-MULTFRC’s 
utilization  is  only  77%.  Therefore,  by  avoiding  the  “quantization  effect”,  E-AIOTFRC 
achieves  better  throughput  performance  than  E-MULTFRC. 

To  examine  the  dynamics  of  E-AIOTFRC  systems,  we  show  throughput,  the  number  of 
virtual  connections  n,  packet  loss  rate,  and  average  RTT  as  a  function  of  time  for  pw  =  0.04 
in  Figure  5.26.  As  seen,  the  throughput  and  the  number  of  connections  are  quite  stable; 

1()Note  the  RTT  for  pw  =  0  is  not  shown  in  Figure  5.25  because  it  represents  the  channel  error  free  case 
in  which  E-MULTFRC  reduces  to  one  TFRC  connection. 
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as  expected,  packet  loss  rate  is  around  0.04  and  RTT  is  properly  controlled  to  be  around 
7 RTT_min.  Similar  results  are  obtained  for  other  values  of  pw. 


To  investigate  the  fairness  of  E-AIOTFRC,  we  carry  out  NS-2  simulations  based  on  the 
“dumbbell”  topology  shown  in  Figure  5.6.  Senders  are  denoted  by  si,i  =  1,...,16,  and 
receivers  are  denoted  by  di,i  =  1 . . . ,  16.  We  investigate  two  types  of  fairness:  the  inter¬ 
protocol  fairness  between  E-AIOTFRC  and  TCP,  and  the  intra-protocol  fairness  within 
E-AIOTFRC. 

The  intra-protocol  fairness  is  defined  as  the  fairness  between  E-AIOTFRC  flows.  In  our 
simulations,  we  run  E-AIOTFRC  on  all  16  sender-receiver  pairs  shown  in  Figure  5.6  for 
5000  seconds,  and  compare  their  throughput.  E-AIOTFRC  is  said  to  be  intra-protocol  fair 
if  all  receivers  get  the  same  throughput.  The  fairness  ratios  for  pw  =  0.01  and  pw  =  0.04  are 
shown  in  Table  5.12.  The  fairness  ratio  is  defined  as  receivers’  throughput  divided  by  the 
average  throughput;  the  closer  to  one,  the  more  fair  the  E-AIOTFRC  system  is.  As  seen, 
the  fairness  ratio  is  fairly  close  to  one,  indicating  E-AIOTFRC  flows  are  fair  to  each  other, 
at  least  in  this  simulation  setting.  The  bandwidth  utilization  ratios  are  95%  for  pw  =  0.01 
and  98%  for  pw  =  0.04. 


Table  5.12.  Simulation  results  for  intra-protocol  fairness  of  E-AIOTFRC. 


receiver 

fairness 

ratio 

p.u,=0.01 

fairness 

ratio 
p„,= 0.04 

receiver 

fairness 

ratio 

p™=0.01 

fairness 

ratio 
77,,= 0.04 

dl 

1.00 

0.99 

d9 

1.02 

1.03 

d2 

0.99 

0.99 
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The  inter-protocol  fairness  is  defined  as  the  fairness  between  E-AIOTFRC  and  TCP11. 
In  our  simulations,  we  run  E-AIOTFRC  on  the  first  8  sender- receiver  pairs,  i.e.  (si,  di ),  i  = 
1, . . . ,  8,  and  TCP  on  the  remaining  8  sender-receiver  pairs  shown  in  Figure  5.6;  each  session 
xlWe  use  TCP  SACK  implementation  in  simulations. 
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lasts  5000  seconds,  and  we  compare  their  throughput  for  pw  =  0.01  and  pw  =  0.02.  Under 
the  simulation  settings,  each  E-AIOTFRC  consumes  more  bandwidth  than  one  TCP  under 
full  utilization.  This  is  because  in  this  case,  the  wireless  channel  error  rate  is  large  enough 
to  make  the  number  of  virtual  connections  of  each  E-AIOTFRC  to  be  larger  than  one. 
Hence,  it  is  meaningless  to  define  the  fairness  between  E-AIOTFRC  and  TCP  as  having  the 
same  throughput.12  As  such,  in  our  simulations,  we  define  E-AIOTFRC  to  be  fair  to  TCP 
if  it  does  not  result  in  a  decrease  in  TCP’s  throughput.  Specifically  for  our  simulations, 
it  implies  TCP  retains  the  same  throughput  whether  or  not  it  coexists  with  E-AIOTFRC 
under  the  same  network  setting.  The  throughput  of  E-AIOTFRC  and  TCP,  as  well  as  the 
total  bandwidth  utilization  ratios  for  the  setup  shown  in  Figure  5.6,  are  shown  in  Table 
5.13  for  two  scenarios:  (a)  8  E-AIOTFRC  coexisting  with  8  TCP  connections,  (b)  16  TCP 
connections. 

Figures  5.27  and  5.28  also  show  the  dynamics  of  throughput,  packet  loss  rate,  RTT, 
and  the  number  of  virtual  connections  n.  Comparing  E-AIOTFRC+TCP  with  TCP-alone, 
we  see  the  former  has  a  much  higher  utilization  of  the  wireless  bandwidth  at  the  expense  of 
lower  TCP  throughput.  A  careful  examination  of  Figure  5.27  reveals  that  this  throughput 
drop  is  caused  by  the  higher  RTT  for  E-AIOTFRC+TCP  as  compared  with  TCP-alone. 
For  example,  for  pw  =  0.01  and  7  =  0.5,  E-AIOTFRC+TCP  experiences  around  0.6  seconds 
RTT,  while  TCP-alone  only  experiences  0.5  seconds  RTT,  i.e.  the  propagation  delay.  As 
TCP’s  throughput  is  known  to  be  inversely  proportional  to  RTT,  the  20%  increase  in  the 
RTT  explains  the  16%  decrease  in  the  TCP’s  throughput  shown  in  first  row  in  Table  5.13. 

This  increase  in  the  RTT  is,  by  design,  a  consequence  of  E-AIOTFRC  controlling  the 
number  of  virtual  connections  n  according  to  Equation  (4.4).  As  n  is  only  decreased  after 
the  queuing  delay  exceeds  the  threshold  - yrttjmin ,  round  trip  time  is  increased  when  E- 
AIOTFRC  increases  n  to  achieve  full  utilization.  One  way  to  address  this  problem  is  to 
use  a  smaller  value  for  7,  in  order  to  reduce  the  increase  in  the  RTT,  and  hence  minimize 
the  TCP’s  throughput  drop.  However,  smaller  values  of  7  also  results  in  lower  bandwidth 

12Obviously,  there  are  situations  in  which  E-AIOTFRC  ends  up  with  performing  similar  to  one  TFRC. 
An  example  would  be  E-AIOTFRC  competing  for  bandwidth  with  TCP  on  wired  networks.  In  that  case, 
however,  the  fairness  between  E-AIOTFRC  and  TCP  is  reduced  to  the  fairness  between  TFRC  and  TCP, 
and  has  been  well  explored  in  [23]. 
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Table  5.13. 

Simulation  results  for  fairness  between  ’ 

E-AIOTFRC  and  TCP 

settings 

8  E-AIOTFRC  +  8  TCP 

16  TCP 

ave.  thput. 
(E-AIOTFRC) 
(kbps) 

ave.  thput. 
(TCP) 
(kbps) 

utili¬ 

zation 

(%) 

ave.  thput. 
(TCP) 
(kbps) 

utili¬ 

zation 

(%) 

pw= 0.01 

7=0.5 

436.501 

168.048 

98 

200.168 

65 

pw= 0.01 
7=0.1 

379.656 

185.286 

91 

200.168 

65 

pw= 0.02 

7=0.5 

486.821 

120.313 

99 

139.674 

46 

pw= 0.02 
7=0.1 

449.226 

130.953 

95 

139.674 

46 

utilization  due  to  increased  sensitivity  of  E-AIOTFRC  to  fluctuations  in  measured  RTT. 
As  shown  in  Table  5.13,  7  =  0.1  results  in  a  smaller  drop  in  the  TCP’s  throughput  than 
7  =  0.5. 


5.5  Video  Related  Simulations 

To  evaluate  the  performance  of  E-MULTFRC  in  video  streaming  applications,  we  sim¬ 
ulate  streaming  of  a  60  second  long  video  clip  through  a  channel,  with  throughput  trace 
corresponding  to  one  of  the  traces  obtained  from  actual  experiments  over  lxRTT  CDMA  as 
described  in  Section  5.2.2.  Our  goal  is  to  compare  the  quality  of  video  streaming  achievable 
using  one  TFRC  connection  with  that  of  E-MULTFRC. 

We  encode  300  frames  of  news.cif  sequence  using  MPEG-4  at  bit  rates  varying  from 
50kps  to  100  kbps  13  as  controlled  by  TMN-5  [49].  The  frame  rate  is  10  frame  per  second;  the 
I- frame  refresh  rate  is  once  every  fifteen  frames.  The  coded  video  bit  stream  is  packetized 
with  fixed  packet  size  of  760  bytes.  The  packets  are  then  protected  using  Reed-Solomon 
(RS)  codes  with  different  protection  levels  for  one  TFRC  and  E-MULTFRC.  This  is  because 
packet  loss  statistics  are  different  in  the  two  cases.  Specifically,  the  statistics  of  30  minutes 
long  trace  indicates  the  longest  burst  loss  to  be  6  packets  long  for  one  TFRC  and  11  packets 

13  Our  choices  of  video  bit  rates  are  related  to  the  available  bandwidth  in  today’s  commercial  wireless 
networks. 
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long  for  E-MULTFRC.  Thus,  we  apply  RS(56,50)  to  one  TFRC  case,  and  RS(61,50)  to  E- 
MULTFRC  case  in  order  to  sufficiently  protect  packets  in  both  cases.  Under  ideal  conditions 
where  all  packets  in  both  schemes  get  through,  the  decoded  video  quality  is  identical  between 
the  two  schemes.  This  is  because  before  adding  RS  code,  the  source  video  bit  rate  is  chosen 
to  be  the  same  for  both  schemes. 

The  RS-coded  packets  are  then  passed  through  channels  simulated  using  one  TFRC, 
and  E-MULTFRC  packet  level  traces  each  lasting  70  seconds,  selected  from  the  30  minutes 
long  actual  experiments  described  in  Section  5.2.  The  throughput  and  packet  loss  details  for 
a  70  second  long  segment  of  one  TFRC  and  E-MULTFRC  connections  are  shown  in  Figure 
5.29.  As  Seen,  both  throughput  and  packet  loss  rate  are  higher  for  E-MULTFRC  than 
for  one  TFRC  case.  The  large  throughput  fluctuations  in  E-MULTFRC  due  to  changing 
number  of  connections  can  potentially  be  argued  not  to  be  suitable  for  video  applications 
in  general;  however,  proper  buffering  can  absorb  these  fluctuations  in  non-delay  sensitive 
streaming  applications. 

The  receiver  decodes  the  received  RS-coded  packets  and  stores  the  MPEG-4  bit  streams 
into  a  playback  buffer.  In  this  simulation,  we  fill  the  buffer  with  10  seconds  worth  of  data 
before  starting  the  MPEG-4  decode  and  display  process.  The  playback  rate  is  fixed  at  10 
frames  per  second,  and  hence  decoding  process  is  stopped  and  the  display  is  frozen  whenever 
the  playback  buffer  is  empty. 

To  show  the  efficiency  of  E-MULTFRC,  we  compare  the  playback  buffer  occupancies 
of  E-MULTFRC  and  one  TFRC  for  several  bit  rates  in  Figure  5.30.  At  each  bit  rate  the 
source  rate  is  the  same  for  both  E-MULTFRC  and  TFRC,  but  the  total  streaming  bit  rate 
after  FEC  is  higher  for  E-MULTFRC  than  TFRC.  Thus,  if  both  streams  were  received 
successfully  at  the  receiver,  we  would  expect  the  video  quality  to  be  identical.  As  seen  in 
Figure  5.30,  compared  to  one  TFRC  case,  E-MULTFRC  can  sustain  video  streaming  at 
higher  bit  rates  and  hence  higher  visual  quality,  despite  the  fact  that  it  needs  stronger  FEC 
to  combat  the  higher  packet  loss  rate.  Specifically,  E-MULTFRC  can  sustain  source  bit 
rate  of  90kbps  and  total  streaming  rate,  including  FEC,  of  110kbps;  while  TFRC  can  only 
sustain  some  bit  rate  of  50kbps  and  total  streaming  rate,  including  FEC,  of  56  kbps. 
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Figure  5.25.  NS-2  simulations  of  E-AIOTFRC  for  Bw  =  1  Mbps  and  RTTmin  =  168  ms;  (a) 
throughput,  (b)  end-to-end  packet  loss  rate,  (c)  end-to-end  RTT,  (d)  number  of  connections, 
all  as  a  function  of  packet  error  rate  on  the  wireless  channel. 
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Figure  5.26.  NS-2  simulations  of  E-AIOTFRC  for  Bw  =  1  Mbps  and  pw  =  0.04; 
(a) throughput,  (b)  number  of  connections  ,  (c)  end-to-end  packet  loss  rate,  (d)  end-to-end 
RTT,  all  as  a  function  of  time. 
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Figure  5.27.  NS-2  simulation  results  of  E-AIOTFRC  coexisting  with  TCP,  for  the  case 
pw  =  0.01,7  =  0.5:  the  dynamics  of  (a) throughput,  (b)  number  of  connections  ,  (c)  end-to- 
end  packet  loss  rate,  (d)  end-to-end  RTT,  all  as  a  function  of  time. 
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Figure  5.28.  NS-2  simulation  results  of  E-AIOTFRC  coexisting  with  TCP,  for  the  case 
pw  =  0.01,7  =  0.1:  (a)throughput,  (b)  number  of  connections  ,  (c)  end-to-end  packet  loss 
rate,  (d)  end-to-end  RTT,  all  as  a  function  of  time. 


120 


loss  throughput  (bps) 


(a) 


(b) 


Figure  5.29.  Throughput  and  packet  loss  details  for  (a)  one  TFRC;  (b)  E-MULTFRC. 
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Figure  5.30.  Throughput  and  packet  loss  details  for  one  TFRC  (left)  and  E-MULTFRC 
(right):  the  source  bit  rate  is  at  (a)  50kbps;  (b)  70kbps;  (c)  90kbps. 
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Chapter  6 


Conclusion  and  Future  Work 


This  chapter  includes  concluding  remarks  and  future  work. 


6.1  Discussion 

E-MULTCP,  E-MULTFRC,  MULTFRC,  and  E-AIOTFRC  are  different  from  existing 
schemes  such  as  TCP- Vegas  [13]  and  VTP  [56]  which  use  delay  or  round  trip  time  variation 
to  infer  congestions  in  several  ways:  first  both  TCP- Vegas  and  VTP  are  transport  layer 
solutions,  and  could  solve  the  wireless  problem  only  if  it  were  deployed  universally  by  every 
single  computer,  cell  phone,  PDA,  and  handheld  device.  But  the  fact  is  that  there  are 
many  flavors  of  TCP  deployed  today  on  a  heterogeneous  set  of  devices,  and  trying  to  make 
everyone  use  a  new  version  of  TCP  is  impractical,  if  not  impossible.  As  such,  the  strength 
of  our  proposed  application  layer  scheme  is  that  it  requires  absolutely  no  modification  to 
the  transport  layer  protocol  stack,  and  hence  no  modifications  to  the  operating  systems  of 
the  end  user  devices.  It  also  does  not  require  any  changes  to  the  network  infrastructure 
such  as  routers. 

Second,  TCP- Vegas  needs  to  measure  precise  queueing  delay  every  RTT,  e.g.  typically 
on  the  order  of  tens  of  milliseconds.  It  is  difficult  to  measure  this  accurately  in  such  a  short 
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interval,  because  the  large  variance  in  the  packet  processing  time  at  routers  and  end-hosts 
may  overwhelm  the  actual  value  of  queuing  delay.  In  contrast,  our  proposed  scheme  is 
based  on  only  one  bit  of  information  to  detect  existence  of  queuing  delay  once  in  a  while, 
e.g.  every  20  seconds  in  current  implementation  of  E-MULTCP.  Such  one  bit  of  information 
is  relatively  easy  to  measure  accurately. 


6.2  Conclusions 

In  this  thesis,  we  have  formulated  flow  control  problem  in  wireless  networks  as  a  general 
concave  optimization  problem,  of  which  Kelly’s  optimization  problem  can  be  shown  to  be  a 
special  case.  This  reformulation  results  in  a  new  class  of  end-to-end  based  solutions,  in  which 
an  appropriate  number  of  connections  are  opened  at  the  application  layer  by  the  sender. 
The  solutions  require  only  one  bit  of  information  on  whether  or  not  the  route  is  congested, 
making  it  easy  to  estimate  accurately  at  the  application  layer.  Hence  no  modification  to 
either  existing  transport  protocols  stack,  for  example  TCP,  or  infrastructure  elements,  for 
example  routers,  is  needed.  We  have  shown  that  our  proposed  scheme  has  a  unique  stable 
equilibrium  that  solves  the  concave  optimization  problem,  implying  the  stability,  scalability 
and  optimality  of  the  solution.  Specifically,  the  unique  optimal  equilibrium  is  semi-globally 
exponentially  stable.  Stability,  optimality  and  scalability  of  proposed  solution  coexisting 
with  TCP  can  also  be  inferred  in  a  straightforward  fashion. 

We  have  also  shown  that  our  proposed  solution  follows  a  general  framework  for  flow 
control.  In  this  framework,  solving  a  flow  control  problem  is  interpreted  as  pursuing  a 
particular  equilibrium  of  users’  rates.  It  is  sufficient  to  achieve  this  goal  by  using  one 
control  law  for  the  number  of  connections  in  a  fast  timescale,  and  another  control  law  for 
sending  rate  of  individual  connection  in  a  slow  timescale.  If  both  control  laws  can  guarantee 
exponential  convergence  to  selected  equilibria,  then  in  general  users’  rates  will  converge  to 
the  desired  equilibrium  exponentially.  A  significant  advantage  of  this  framework  is  that 
it  is  possible  to  modify  a  control  law  in  one  timescale  without  affecting  the  one  in  the 
other  timescale,  while  still  maintaining  convergence  properties.  Following  this  framework, 
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we  have  easily  derived  a  variant  of  proposed  solution  with  similar  convergence  performance 
guarantees. 

In  practice,  these  results  guarantee  all  network  bottlenecks  to  be  fully  utilized,  yet  the 
network  is  free  of  congestion  collapse,  for  arbitrary  topology,  arbitrary  initial  sending  rates, 
and  arbitrary  number  of  users  applying  either  proposed  solution  or  TCP.  Furthermore, 
users’  rates  globally  exponentially  converge  to  a  unique  and  optimal  equilibrium,  resulting 
in  no  rate  fluctuation  in  stationary  state. 

To  demonstrate  the  generality  of  our  proposed  solution,  we  have  developed  practical 
scheme  E-MULTCP  for  data  transmission  over  wireless  network,  and  E-MULTFRC,  MULT- 
FRC,  and  E-AIOTFRC  for  streaming  over  wireless  networks.  Among  them,  E-MULTCP, 
E-MULTFRC,  and  E-AIOTFRC  use  HMD  control  law,  and  are  designed  based  on  the  pro¬ 
posed  solution;  MULTFRC  uses  HAD  control  law,  and  is  designed  based  on  the  variant  of 
proposed  solution.  Their  efficient  performance,  and  fairness  to  both  themselves  and  TCP 
are  characterized  and  evaluated  using  NS-2  simulations,  and  experiments  over  commer¬ 
cial  lxRTT  and  EVDO  wireless  networks.  Analysis  and  simulation  results  indicate  these 
schemes  in  fact  work  in  both  wired  and  wireless  scenarios,  and  result  in  anywhere  between 
25%  and  65%  throughput  improvement  over  TCP  or  TFRC  in  commercial  cellular  wireless 
networks. 

In  addition,  Lemma  3.3.4  states  that  for  equilibrium  of  continuous  systems,  local  ex¬ 
ponential  stability  and  global  asymptotical  stability  indicate  the  senri-global  exponential 
stability.  We  believe  that  this  lemma  is  general  and  as  such  can  be  applied  to  other  prob¬ 
lems  in  control  for  example.  We  also  believe  that  our  functions  for  applying  the  La  Salle 
principle,  in  the  proof  of  Lemma  3.3.3,  may  provide  new  insight  to  searching  Lyapunov 
functions,  or  functions  for  the  La  Salle  principle  when  exploring  the  stability  of  general 
nonlinear  systems. 

In  summary,  we  have  provided  promising  answers  to  the  two  questions  we  set  out  to 
answer  in  the  Chapter  1.1  of  this  thesis: 

Question:  Is  it  possible  to  solve  the  problem  of  flow  control  in  wireless  networks,  without 
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changing  today’s  network  infrastructure,  operating  systems,  protocol  stack,  or  violating  the 
end-to-end  principle? 

It  is  indeed  possible  to  use  the  application  layer  to  solve  this  problem.  Several  schemes 
proposed  in  this  thesis,  such  as  E-MULTCP,  E-MULTFRC,  MULTFRC  and  E-AIOTFRC, 
are  concrete  examples  of  such  possibility,  with  theoretical  and  experimental  verifications. 

Question:  What  is  the  framework  for  flow  control  problem  in  wired  or  wireless  networks, 
without  modifying  today ’s  network  infrastructure,  operating  systems,  or  the  protocol  stacks  ? 

In  the  general  framework  proposed  in  Section  3.6,  rate  of  individual  connections  and 
the  number  of  connections  opened  by  a  user  are  controlled  independently  in  two  separate 
timescales.  Hence,  it  is  possible,  but  not  necessary,  to  use  an  existing  control  law  such  as 
TCP  for  rate  of  individual  connections,  and  to  design  new  control  laws  for  the  number  of 
connections;  this  results  in  no  modifications  to  transport  layer  protocol  stacks  and  network 
infrastructure  elements. 


6.3  Future  Work 

Even  though  Cj  and  e3  are  assumed  to  be  constant  in  our  analysis  in  Chapter  3,  in  some 
networks  such  as  wireless  Local  Area  Networks  (WLAN)  and  CDMA  networks,  Cj  and  fa¬ 
nlight  be  time  varying  or  even  change  in  a  correlated  fashion.  E-MULTCP,  E-MULTFRC, 
MULTFRC,  and  E-AIOTFRC  adjust  the  number  of  connections  in  a  timescale  much  slower 
than  that  of  each  connection’s  sending  rate,  in  order  to  satisfy  the  two  timescale  assumption, 
as  well  as  to  reduce  the  complexity  and  overhead  of  opening/closing  connections.  As  a 
result,  in  a  scenario  where  Cj  and  Cj  are  time  varying,  by  design,  they  could  not  react 
instantaneously.  Hence,  it  is  interesting  to  quantify  how  fast  they  can  adapt  to  the  changes 
in  Cj  and  Cj.  In  practice,  our  experimental  results  in  this  thesis  have  verified  that  our 
proposed  schemes  work  in  a  cemmercial  CDMA  network,  where  the  Cj  and  e3  are  typically 
not  constant. 

Currently,  E-MULTCP,  E-MULTFRC,  MULTFRC,  and  E-AIOTFRC  rely  on  accurate 
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detection  of  queuing  delay  along  the  route,  which  in  turn  requires  knowledge  of  the  propa¬ 
gation  delay.  In  situation  where  propagation  delay  is  highly  volatile,  such  as  in  3G  wireless 
networks  that  deploy  opportunistic  scheduling,  their  performance  could  potentially  degrade 
due  to  inaccurate  estimation  of  queuing  delay.  As  discussed  in  the  beginning  of  Section  5.1.2, 
even  though  it  is  possible  to  choose  a  large  7  to  tolerate  inaccuracy  in  estimating  queuing 
delay,  it  would  be  desirable  to  develop  a  more  reliable  way  to  estimate  the  congestion  status 
of  route,  i.e.  the  one  bit  of  information  needed  by  these  practical  schemes. 

There  are  many  other  interesting  and  important  directions  for  future  research.  Stability 
in  the  presence  of  delay  and  noisy  disturbances  are  of  great  interest,  from  both  control  and 
networking  points  of  view.  The  fairness  properties  of  the  equilibrium  rates  are  also  of 
interest. 

Our  framework  captures  the  impact  of  wireless  channel  packet  loss  on  TCP  and  TFRC 
performance  over  wireless.  However,  it  does  not  model  the  timeout  effect  of  TCP  and 
TFRC  over  wireless  possibly  caused  by  link  layer  local  retransmissions  or  heavy  wireless 
loss,  potentially  degrading  the  performance.  It  is  desirable  to  extend  our  framework  in  order 
to  take  into  account  the  effect  of  timeout  in  wireless  networks. 

In  our  current  framework,  we  assume  one  data  transmission  consists  of  two  end-hosts 
and  only  one  path  between  them.  In  some  scenarios,  using  multiple  paths  to  transmit  data 
between  two  end-hosts  has  pronounced  advantages  in  terms  of  throughput  and  scalability, 
shown  in  many  practical  peer-to-peer  applications  and  wireless  multipath  communications. 
Multipath  communication  allows  the  sending  rate  of  a  user  to  be  a  sum  of  the  sending  rates 
along  two  or  more  paths.  It  is  not  clear  how  to  define  proper  fairness  among  users  and  hence 
is  not  clear  how  the  proposed  solution  E-MULTCP  and  the  framework  can  be  applied  to 
this  scenario. 

Last  but  not  the  least,  it  is  interesting  to  examine  whether  it  is  possible  to  use  a  different 
utility  maximization  problem  that  leads  to  another  fundamentally  different  solution  for  the 
TCP  and  TFRC  over  wireless  network  problem. 
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Appendix  A 


Proof  of  Theorem  3.3.1 


Theorem  3.3.1:  For  arbitrary  (3  >  0,  the  approximate  system  in  Equation  (3.14)  has  a 
unique  equilibrium,  denoted  by  (x*,n*),  given  by 


n:  = 


xr  = 


r  £  R: 


V2S 


y/ fClljer  9j{y*j  ))F\l'HjQr\9j(y*j  )+ei] 


r  £  R. 


(A.l) 


Further,  this  unique  equilibrium  solves  the  following  concave  optimization  problem 


max 

x>0 


xr)-^2  9j(z)dz, 
reR  j&J 


(A.2) 


with  Ur  being  concave  function: 


Ur(xr)  =  h. 

Jo 


-l 


2  S2 
ThP 


dv,  r  £  R, 


where  hr  1  is  the  inverse  of  a  monotonically  increasing  function  hr: 

hr(z)=  (^2ej  +  z'\f{z)=  (^2ej+^e  1 

^  j£r  /  ^  jf'Er 


e@z  +  1  ’ 


r  £  R. 


Proof.  First  note  ( x*,n *)  is  the  equilibrium  of  the  system  in  Equation  (3.14),  and  x*  is 
also  the  solution  for  the  optimization  problem  in  Equation  (3.20),  which  can  be  seen  by 
setting  the  derivative  to  zero.  By  the  definition  of  hr(z )  and  gj(z),  it  is  not  difficult  to  see 
the  objective  function  in  Equation  (3.20)  is  a  concave  function.  Note  the  constraint  set  of 
the  optimization  problem  shown  in  Equation  (3.20)  is  convex,  and  that  the  optimization 
problem  is  in  fact  a  concave  optimization  problem.  Hence  it  has  a  unique  solution,  which  is 
in  fact  x*.  Since  the  equilibrium  must  lie  on  the  equilibrium  manifold  in  Equation  (3.17), 
by  Lemma  3.3.1  we  know  the  unique  x*  leads  to  a  unique  n*.  Therefore,  the  equilibrium 
(x*,n*)  exists  and  is  unique.  □ 
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Appendix  B 


Proof  of  theorem  3.3.2 


Theorem  3.3.2:  For  arbitrary  (3  >  0,  under  the  two  timescale  assumption,  the  unique 
equilibrium  of  the  reduced  system  in  Equation  (3.16)  is  locally  exponentially  stable.  Also, 
the  unique  equilibrium  of  the  approximate  system  in  Equation  (3.1  f),  (x*,n*),  is  locally 
exponentially  stable. 

Proof.  Let  G(x)  =  diag(^7^fr-)  and 

D(x)  =  diag{—^2lej  +  9j(yj)]}  +  * ATdiag{g'j(yj)}A , 

r  j&r 

where  A  is  the  connectivity  matrix  defined  in  Section  3.1.  We  can  have  the  relation  between 
x  and  h  on  the  equilibrium  manifold  as: 

G(x)D(x)x  =  diag{nr}h, 

Combining  the  dynamics  of  n  in  the  reduced  system,  we  can  rewrite  the  above  equation  as: 
X  =  cL»'1(x)G“1(®){1  -  + 

j&  j£r 


Around  the  equilibrium  of  the  reduced  system,  let  xr(t)  =  x*  +  zr(t),  denote  D(x*)  as 
D  and  G(x*)  as  G  ;  after  linearization,  we  have  that,  Vr  G  R. 


z(t )  =  -cD-'G-1 


2diag  j  f(^gj{y]))  GD+ 


jSr 


G-diag(J2  (ej +  »($))) 


j&r 


diag  (  f'(^gj(yj))  )  AT  diag(g'j(y*))A 

j£r 


z(t), 


(B.l) 


where  g'j(y*) 


_  Cj  e 


(y*)2 

1+e  yJ 


3C7  >  o,  jeJ. 
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Denote 


E  =  2diag  I  fC^gjiVj))  1  GD+G-diag(^2  (< j  ~  </,(//}))  )diag  (/'(#,(?/*)))  Ardiag(g'j(y*))A. 

\  j£r  J  j£r 

Then  by  simple  linear  algebra  arguments,  the  system  in  Equation  (B.l)  is  stable  if  and  only 
if  D~1G~1E  has  all  positive  eigenvalues.  We  now  show  that  this  requirement  is  verified. 

First  note  that  this  is  equivalent  to  showing  ED~lG~ 1  has  all  eigenvalues  be  positive 
since  ED^G^1  is  similar  to  D^G^1  E.  But 

ED-'G-i  =  2 G-dio9 


■diag 


j£r 

Xr  *  f(Ej&9j^)) 


f'(J2jer9j{yj))[J2jer(ej  +  9j(VjW  J 

}ATdiag(gj(yj))AD~1\  G-1, 


+  2<'“9{e  ierl'i  +  m(m 


Define  the  terms  inside  the  curly  brackets  as  B  for  later  usage.  ATdiag(g'j(yj))AD  1  is  a 
product  of  a  positive  definite  matrix  and  a  (semi)-positive  definite  matrix,  having  all  non¬ 
negative  eigenvalues.  On  the  other  hand,  ATdiag(g'j(y*))A  =  2 (D  —  diag^3^3^93^3^  }); 


hence 


ATdiag{g'j{y*))AD  1  =  2  •  diag{  ^  9j(yVj  ^  } 


xl 


diag{ 


Xj&[*j  +  9jte)\ 


}~D 


-l 


Now  diag{j*~ — [*r+y{y*)\ }  —  B  1  E  0.  This  can  be  shown  by  contradiction.  Suppose 

the  contrary;  left  multiple  the  above  equation  by  (diag{  ^  ,  and  right 

Ejerh+gjOj)]  ~A1//2. 


multiple  the  above  equation  by 


;  then  the  left  hand  side  has  all 


nonnegative  eigenvalues,  while  the  right  hand  side  is  a  non-semi-positive  definite  matrix 
with  at  least  one  eigenvalue  being  negative,  resulting  in  contradiction. 

We  claim  ED^G-1  has  all  positive  eigenvalues,  due  to  the  following  three  facts: 


By  0  since  it  is  a  sum  of  two  positive  definite  matrices. 


diag  (  /'(E  j£r  9j(y*i )) 


Ejerfa+gjfaj)]2 


Jjer 

product  of  two  positive  definite  matrices; 


B  has  all  positive  eigenvalues,  because  it  is  the 


ED  LG  1  has  all  positive  eigenvalues,  because  it  is  similar  to 
diag  (/'{>_:  .  ,. //.(//J))  -''  I  B. 
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Eventually,  D  1 G  1E  has  all  positive  eigenvalues  and  hence  the  reduced  system  in 
Equation  (B.l)  is  locally  exponentially  stable  for  arbitrary  (3  >  0. 

Hence  combining  the  fact  that  the  boundary  system  is  locally  exponentially  stable,  we 
conclude  the  entire  system  is  locally  exponentially  stable,  according  to  Theorem  11.4  in 
[30].  □ 
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Appendix  C 


Proof  of  Lemma  3.3.2 


Lemma  3.3.2:  There  exists  a  compact  set,  denoted  by  Hi,  for  n(t )  in  the  reduced  system  in 
Equation  (3.16)  with  arbitrary  [3  >  0,  such  that  any  compact  set  containing  it  is  a  positively 
invariant  one.  A  positive  invariant  set  is  a  set  with  all  trajectories  on  its  boundary  pointing 
inwards;  as  such,  no  trajectories  inside  the  set  will  ever  move  out.  The  same  observation 
is  also  true  for  x(t)  in  the  boundary  layer  system  Equation  (3.15),  and  the  corresponding 
compact  set  is  defined  as  ^(n),  a  function  of  n. 


Proof.  To  construct  the  compact  set  Hi  for  wr(t)  in  the  reduced  system,  first  note 
a)  if  route  r  is  not  congested,  then  gj(yj(t ))  <  ^ln2;  so 


xr(t)  > 


nr(t)S 

Tr  2) 


As  nr(t)  increases,  xr(t)  will  eventually  hit  minjer  Cj  and  route  r  becomes  congested  (cross 
traffic  only  helps  to  cause  congestion).  Hence,  if  nr(t)  is  sufficiently  large,  route  r  will  be 
congested. 

b)  if  route  r  is  congested,  then  we  must  have  Yljer  >  4  In  2.  Together  with  the 

fact  f (Yljer  9j(yj(t)))  a  nondecreasing  function  of  9j{Uj{t)),  we  have 


fQ29j(yj(t)))  > 


j&r 

Hence  as  long  as  route  r  is  congested,  we  have 

/ 

1 


eln  2  _  1 

+  eln2  =  3 ' 


hr(t )  =  c 


nr(t ) 


Mt)fC^2gj{yj{t)))\  <c 

j£r  / 


“  \nAt) 


There  must  exist  one  such  that  nfiax{t)  <  0. 

Therefore,  there  exists  large  enough  n™ax  such  that  route  r  is  congested  regardless 
of  cross  traffic,  i.e.  even  only  user  r  is  active  and  others  are  inactive,  and  rdfLax{t)  <  0. 
Eventually,  Hi  can  be  defined  as 


Hi  =  [0,  n" 


max  j 


x  [0,  n^ax] 


[0,n% 


max  j 
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It  is  easy  to  check  the  flow  of  the  vector  field  satisfies: 


•  if  nr(t)  <  0,  rir(t)  >  0  according  to  Equation  (3.16); 

•  if  nr(t)  >  n™ax,  then  by  n™“x,s  definition,  rir(t)  <  0. 

Therefore,  on  the  boundary  of  any  compact  set  containing  Hi,  vector  field  defined  in  Equa¬ 
tion  (3.16)  points  inward.  Hence  by  definition,  the  compact  set  is  positive  invariant. 

Similarly,  a  compact  set  fi2(ra)  can  be  defined  for  x(t)  in  boundary  layer  system  in 
Equation  (3.15),  as  follows: 

n2(n)  =  [0 lXrx{ni)]  x  [0,xrx(n2)]---[0,xTx(nN)}, 

where  x!.na3:(nr)  =  — —  Se  ^ s  ,  satisfies  that  given  nr  and  regardless  of  cross  traffics 

along  route  r.  If  xr(t )  >  x™ax(nr),  then  xr(t)  <  0. 

From  networking  point  of  view,  containing  Hi  implies  the  number  of  connections  of  each 
user  should  be  allowed  to  take  large  enough  values  to  fully  utilize  his  bottleneck,  even  if  only 
he  is  active.  Similarly,  containing  H2(n)  means  sending  rate  of  each  user  can  be  sufficiently 
large  to  achieve  its  equilibrium.  These  requirements  are  trivial  to  meet  in  practice.  □ 
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Appendix  D 


Proof  of  Lemma  3.3.3 


Lemma  3.3.3:  The  unique  equilibrium  of  reduced  layer  system  in  Equation  (3.16),  with 
arbitrary  /3  >  0,  is  a  globally  asymptotically  stable  one. 

Proof.  Define  zr  =  4J^r ,  and  pr  =  for  r  £  R.  We  have 


2  prn(. 


Z  =  diag( — ^-)diag(nr)h  —  diag( — M) 


i-D.--  MPr 


D  ldiag(—^-)diag{nr){n) 

=  diag(nr^/zf)diag(-^-)Adiag(-^-)diag(nr)h 


(D.l) 


where  A  =  diag(xf/2n(pr)  —  D  1  has  been  shown  in  Appendix  B  to  be  a  semi-positive 
definite  matrix. 


Define  nondecreasing  functions  <f>r(y)  and  (pr(y),r  £  R  as  follows: 

r/-1(  D+O-  ^ 


— dy,  r  £  i? 

Vy 


Mv)  =  J 

My)  = 


Vv 


dy,  r  £  R 


(D.2) 

(D.3) 


The  following  is  a  La  Salle  function  for  the  reduced  system: 

fZr  ^ 


V (n,  z)  =  —  ^2  cr 

r£R 


n. 


Vyn  i 


dy  -  /  —pf(y  -  Mdy 

Jer  Vy 


1,1 


+  /  ~2 MM)dy+  /  MM)dy 


i  o  y  y 


i  o  r 
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Its  Lee  derivative  is 


V(n,z)  =  - p^f(zr  -  er)  )  zr  - 

'  \  yfzrnr  y/zrnr 


r£R 


E' 


nr 


r>  L 

r&R  r  L 


$r(  9)  $r(f(zr  ^r)) 

nz. 


~  Vr{f(zr  ~  €r)) 


n~ 


'y  ]  crnr 

r&R 

-hdiag(nr)diag(-^-)  A  diag(-^-)diag(nr)h  — 


£■ 


nr 


r>  L 

reR  r  L 


( Pr{  9)  $r(f(zr  ^r)) 

ni 


y  ]  crnr 

reR 

<  0. 


^(^2)  -  Vr{f(zr  -  er)) 


nt 


(D.4) 


The  equality  is  taken  only  when  h  =  0,  i.e.  at  the  equilibrium.  Therefore,  the  system 
is  globally  asymptotically  stable. 

□ 
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Appendix  E 


Proof  of  Lemma  3.3.4 


Lemma  3.3.4:  Consider  a  system  £  =  </?(£,  t,  £o)  satisfying  the  following  assumptions: 

•  it  has  a  unique  equilibrium  at  0  that  is  locally  exponentially  stable  and  globally  asymp¬ 
totically  stable; 

•  ¥>(£)  t,  £o)  is  continuous. 

Then  the  equilibrium  of  the  system  is  semi-globally  exponentially  stable. 

Proof.  By  definition  of  local  exponential  stability,  there  exists  a  r  >  0  s.t. 

\m,m  <m0\\e-^,  vh&ii  <*•, 

where  K,  r  and  7  >  0  are  constant.  Without  loss  of  generality,  we  assume  K  >  1  and  r  <  1. 

Define  Tr(£o)  =  inf{i  >  0  :  ||£(i,£o)||  <  r}.  By  definition  of  global  asymptotical  stability, 

L-(£ 0)  <  00. 

Since  £(t,  £ 0)  is  continuous  with  respect  to  t,  let  Mr(£ 0)  =  max{A,  £(t,  £0)  :  0  <  t  < 
Tr}  <  00. 

Define  L  =  Mr(£0)e^^°)/r,  we  claim 

l|{(«.{o)ll<i||{ol|e-7'.  (E.1) 


To  see  this,  we  study  two  cases: 

•  if  ||£o||  <  r,  then  rr(£0)  =  0,  by  setting,  we  already  have  ||£(f,£o)||  <  A'||£0||e^7t  < 

•  if  ||£o||  >  r,  the  following  is  true  for  t  6  [0, Tr(£o)]: 

L||£0||e-7t  =  Mr(£  0)M4e7T’-(«o)e-7t 
>  Mr(£0)  >  ||£(t,£o)||. 

For  t  >  Tr(£ 0),  let  t'  =  t  —  Tr,  we  have  ||£(f'  =  0) ||  <  r.  Applying  the  first  case’s  result 
concludes  our  claim. 
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Therefore,  by  Definition  5.10  in  [46],  Equation  (E.l)  implies  semi-global  exponential  stability 
of  the  equilibrium.  The  senri-global  term  is  due  to  L’s  dependency  on  initial  condition  £q-  D 
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Appendix  F 


Proof  of  Theorem  3.3.3 


Theorem  3.3.3:  The  unique  equilibrium  of  singularly  perturbed  system  in  Equation  (3.14) 
with  arbitrary  (3  >  0  is  semi-globally  exponentially  stable. 

Proof.  We  have  shown  that  the  reduced  system  has  a  unique  equilibrium  (Theorem  3.3.1) 
that  is  globally  asymptotically  stable  (Lemma  3.3.3)  and  locally  exponentially  stable  (The¬ 
orem  3.3.2).  Hence  by  Lemma  3.3.4,  we  conclude  the  unique  equilibrium  of  reduced  system 
in  Equation  (3.16)  is  semi-globally  exponentially  stable. 

Similar  arguments  are  also  true  for  boundary  layer  system.  It  has  a  unique  equilibrium 
that  is  globally  asymptotically  stable  and  locally  exponentially  stable  [29].  Hence,  again 
by  Lemma  3.3.4,  we  conclude  the  unique  equilibrium  of  boundary  layer  system  in  Equation 
(3.16)  is  semi-globally  exponentially  stable. 

If  we  constrain  n(t)  to  a  compact  set  containing  Hi,  denoted  by  Si  and  let  = 

max(nr  :  nr  £  Ei),  then  any  compact  set  containing  PL2{nmax)  is  positive  invariant,  following 
the  arguments  in  Appendix  C.  Constrain  x(t)  to  a  compact  set  containing  H2(nmax), 
denoted  by  E2. 

On  Ei  x  E2,  the  equilibrium  of  the  reduced  system  in  Equation  (3.16)  is  exponentially 
stable,  and  the  one  of  the  boundary  layer  system  in  Equation  (3.15)  is  exponential  stable 
uniformly  in  n  (verification  is  the  same  as  in  Appendix  ILF,  [32]). 

Together  with  the  fact  Ei  x  E2  is  a  positive  invariant,  by  Theorem  11.4  in  [30],  we 
conclude  that  the  singularly  perturbed  system  in  Equation  (3.14)  has  a  unique  equilibrium 
that  is  exponentially  stable  in  Ei  x  E2.  Hence,  the  equilibrium  of  singularly  perturbed 
system  in  Equation  (3.14)  is  semi-globally  exponentially  stable.  □ 
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Appendix  G 


Proof  of  Theorem  3.7.1 


Theorem  3.7.1:  For  arbitrary  (3  >  0,  the  approximate  system  in  Equation  (3.24)  has  a 
unique  equilibrium,  denoted  by  (x*,n*)  as 


nz  = 


r  f(T,jer9:(y*))' 


r  £  R; 


V2S 


r,  reR. 


f(52j£r  9j  (V*  ))Tr\/^2jer  [dj  (llj  )  +eJ  ] 

Further,  this  unique  equilibrium  solves  the  following  concave  optimization  problem 


(G.l) 


Vj 


max 

x>0 


E  Ur(*r)  ~  E  /  9j(z)  dz, 


(G.2) 


reR 


with  Ur  being  concave  function: 


fXr  -  (  2 S2  \ 

Ur(xr )  =  hf1  (  ^2^2  )  du,  r  <E  R, 
where  hf1  is  the  inverse  of  a  monotonically  increasing  function  hr: 

hr(z)  ~  (  Ee^'  +  *)f2(z)  =  (  EeJ  +  ^  ^  1 


(G.3) 


j€r 


j£r 


oftz  _|_  l 


r  £  R. 


Proof.  First  note  (x*,n*)  is  the  equilibrium  of  the  system  in  Equation  (3.24),  and  x*  is 
also  the  solution  for  the  optimization  problem  in  Equation  (3.28),  which  can  be  seen  by 
setting  the  derivative  to  zero.  By  the  definition  of  hr(z )  and  gj(z),  it  is  not  difficult  to  see 
the  objective  function  in  Equation  (3.28)  is  a  concave  function.  Note  the  constraint  set  of 
the  optimization  problem  shown  in  Equation  (3.28)  is  convex,  and  that  the  optimization 
problem  is  in  fact  a  concave  optimization  problem.  Hence  it  has  a  unique  solution,  which  is 
in  fact  x*.  Since  the  equilibrium  must  lie  on  the  equilibrium  manifold  in  Equation  (3.17), 
by  Lemma  3.3.1  we  know  the  unique  x*  leads  to  a  unique  n*.  Therefore,  the  equilibrium 
(x*,n*)  exists  and  is  unique.  □ 
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